2024-04-04T20:41:18.496 INFO:root:teuthology version: 0.0.1.dev265+gd7d6e60 2024-04-04T20:41:18.496 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121 --owner scheduled_yuriw@teuthology --name yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121/orig.config.yaml 2024-04-04T20:41:18.534 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-04T20:41:18.623 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121 branch: wip-ceph_test_rados-partial-reads description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7640121' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-ceph_test_rados-partial-reads ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 66deef1598934c3b1ef2f044e380a7d00244cead valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: debuginfo: true flavor: default sha1: 66deef1598934c3b1ef2f044e380a7d00244cead thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-ceph_test_rados-partial-reads sha1: 66deef1598934c3b1ef2f044e380a7d00244cead owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 3679 sha1: 66deef1598934c3b1ef2f044e380a7d00244cead sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-ceph_test_rados-partial-reads suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 66deef1598934c3b1ef2f044e380a7d00244cead targets: smithi106.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO5pKSz3FNac+x2k6GP1zszcrUyY03hsljHicGucMoj4mXJbN0r0LR7iGE+oQekIZlxoMd2ak7lpFdpiLOTaUQQ= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBqwBEST0LdPtiFBL4rbRllWhJoAZHrBC6yTmDPvJRG/iuIvA4abrIenfHOl7SxciGjcSoBIQe6fuYuK6YSMIqA= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d7d6e60867cadf19fbfd7655a551ae7d58060757 timestamp: 2024-04-04_14:50:58 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-04T20:41:18.623 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa; will attempt to use it 2024-04-04T20:41:18.624 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks 2024-04-04T20:41:18.624 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-04T20:41:18.625 INFO:teuthology.task.internal:Checking packages... 2024-04-04T20:41:18.652 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '66deef1598934c3b1ef2f044e380a7d00244cead' 2024-04-04T20:41:18.653 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-04T20:41:18.653 INFO:teuthology.packaging:ref: None 2024-04-04T20:41:18.653 INFO:teuthology.packaging:tag: None 2024-04-04T20:41:18.653 INFO:teuthology.packaging:branch: wip-ceph_test_rados-partial-reads 2024-04-04T20:41:18.653 INFO:teuthology.packaging:sha1: 66deef1598934c3b1ef2f044e380a7d00244cead 2024-04-04T20:41:18.653 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-ceph_test_rados-partial-reads 2024-04-04T20:41:18.890 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1864.g66deef15 2024-04-04T20:41:18.892 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-04T20:41:18.898 INFO:teuthology.task.internal:no buildpackages task found 2024-04-04T20:41:18.898 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-04T20:41:18.905 INFO:teuthology.task.internal:Saving configuration 2024-04-04T20:41:18.929 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-04T20:41:18.936 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-04T20:41:18.963 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-04 20:36:01.496276', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO5pKSz3FNac+x2k6GP1zszcrUyY03hsljHicGucMoj4mXJbN0r0LR7iGE+oQekIZlxoMd2ak7lpFdpiLOTaUQQ='} 2024-04-04T20:41:18.986 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-04 20:36:01.495100', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBqwBEST0LdPtiFBL4rbRllWhJoAZHrBC6yTmDPvJRG/iuIvA4abrIenfHOl7SxciGjcSoBIQe6fuYuK6YSMIqA='} 2024-04-04T20:41:18.986 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-04T20:41:18.993 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-04-04T20:41:18.993 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-04T20:41:18.993 INFO:teuthology.run_tasks:Running task console_log... 2024-04-04T20:41:19.065 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff3cc7fdf70>, signals=[15]) 2024-04-04T20:41:19.065 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-04T20:41:19.074 INFO:teuthology.task.internal:Opening connections... 2024-04-04T20:41:19.074 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi106.front.sepia.ceph.com 2024-04-04T20:41:19.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-04T20:41:19.172 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-04-04T20:41:19.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-04T20:41:19.271 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-04T20:41:19.278 DEBUG:teuthology.orchestra.run.smithi106:> uname -m 2024-04-04T20:41:19.297 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64 2024-04-04T20:41:19.297 DEBUG:teuthology.orchestra.run.smithi106:> cat /etc/os-release 2024-04-04T20:41:19.354 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="CentOS Stream" 2024-04-04T20:41:19.354 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="9" 2024-04-04T20:41:19.354 INFO:teuthology.orchestra.run.smithi106.stdout:ID="centos" 2024-04-04T20:41:19.354 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE="rhel fedora" 2024-04-04T20:41:19.354 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="9" 2024-04-04T20:41:19.354 INFO:teuthology.orchestra.run.smithi106.stdout:PLATFORM_ID="platform:el9" 2024-04-04T20:41:19.354 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-04T20:41:19.354 INFO:teuthology.orchestra.run.smithi106.stdout:ANSI_COLOR="0;31" 2024-04-04T20:41:19.354 INFO:teuthology.orchestra.run.smithi106.stdout:LOGO="fedora-logo-icon" 2024-04-04T20:41:19.354 INFO:teuthology.orchestra.run.smithi106.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-04T20:41:19.354 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://centos.org/" 2024-04-04T20:41:19.354 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-04T20:41:19.355 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-04T20:41:19.355 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-04T20:41:19.355 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2024-04-04T20:41:19.382 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-04-04T20:41:19.400 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-04-04T20:41:19.401 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-04-04T20:41:19.459 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-04-04T20:41:19.460 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-04-04T20:41:19.460 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-04-04T20:41:19.460 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-04-04T20:41:19.460 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-04-04T20:41:19.460 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-04-04T20:41:19.460 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-04T20:41:19.460 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-04-04T20:41:19.460 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-04-04T20:41:19.460 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-04T20:41:19.460 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-04-04T20:41:19.460 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-04T20:41:19.461 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-04T20:41:19.461 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-04T20:41:19.461 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-04-04T20:41:19.480 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-04T20:41:19.495 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-04T20:41:19.523 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-04T20:41:19.524 DEBUG:teuthology.orchestra.run.smithi106:> test '!' -e /home/ubuntu/cephtest 2024-04-04T20:41:19.527 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-04-04T20:41:19.544 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-04T20:41:19.550 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-04T20:41:19.550 DEBUG:teuthology.orchestra.run.smithi106:> test -z $(ls -A /var/lib/ceph) 2024-04-04T20:41:19.583 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-04-04T20:41:19.670 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-04T20:41:19.727 INFO:teuthology.run_tasks:Running task kernel... 2024-04-04T20:41:19.741 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-04T20:41:19.741 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-04T20:41:19.741 DEBUG:teuthology.orchestra.run.smithi106:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-04T20:41:19.742 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-04T20:41:19.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-04T20:41:19.759 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2024-04-04T20:41:19.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-04T20:41:19.760 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-04-04T20:41:19.816 INFO:teuthology.orchestra.run.smithi106.stdout:5.14.0-361.el9.x86_64 2024-04-04T20:41:19.816 INFO:teuthology.task.kernel:Running kernel on smithi106: 5.14.0-361.el9.x86_64 2024-04-04T20:41:19.816 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum install -y kernel 2024-04-04T20:41:19.817 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-361.el9.x86_64 2024-04-04T20:41:19.817 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.14.0-361.el9.x86_64 2024-04-04T20:41:19.818 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum install -y kernel 2024-04-04T20:41:22.209 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS-9 - AppStream 9.6 MB/s | 6.7 MB 00:00 2024-04-04T20:41:22.229 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-04-04T20:41:23.777 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-04-04T20:41:23.863 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-04-04T20:41:24.709 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-04T20:41:25.161 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.0 MB 00:00 2024-04-04T20:41:27.129 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-04-04T20:41:31.240 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 9.7 MB/s | 6.1 MB 00:00 2024-04-04T20:41:32.616 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 16 kB 00:00 2024-04-04T20:41:46.592 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - AppStream 929 kB/s | 19 MB 00:20 2024-04-04T20:41:50.534 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-04-04T20:41:51.290 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:18 2024-04-04T20:41:52.743 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-04-04T20:41:54.188 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-04T20:41:56.257 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-04T20:41:58.522 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-04T20:41:58.522 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-04T20:41:58.625 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:41:58.638 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:41:58.638 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-04T20:41:58.638 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:41:58.638 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:41:58.638 INFO:teuthology.orchestra.run.smithi203.stdout: kernel x86_64 5.14.0-435.el9 CentOS-BaseOS 212 k 2024-04-04T20:41:58.639 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-04T20:41:58.639 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core x86_64 5.14.0-435.el9 CentOS-BaseOS 15 M 2024-04-04T20:41:58.639 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules x86_64 5.14.0-435.el9 CentOS-BaseOS 34 M 2024-04-04T20:41:58.639 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core x86_64 5.14.0-435.el9 CentOS-BaseOS 28 M 2024-04-04T20:41:58.639 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:41:58.639 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:41:58.639 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:41:58.639 INFO:teuthology.orchestra.run.smithi203.stdout:Install 4 Packages 2024-04-04T20:41:58.639 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:41:58.639 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 77 M 2024-04-04T20:41:58.640 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 123 M 2024-04-04T20:41:58.640 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:41:58.800 INFO:teuthology.orchestra.run.smithi203.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 1.3 MB/s | 212 kB 00:00 2024-04-04T20:41:59.137 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-04T20:41:59.350 INFO:teuthology.orchestra.run.smithi203.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-04-04T20:42:00.017 INFO:teuthology.orchestra.run.smithi203.stdout:(3/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-04-04T20:42:00.434 INFO:teuthology.orchestra.run.smithi203.stdout:(4/4): kernel-modules-core-5.14.0-435.el9.x86_6 17 MB/s | 28 MB 00:01 2024-04-04T20:42:00.436 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:42:00.437 INFO:teuthology.orchestra.run.smithi203.stdout:Total 43 MB/s | 77 MB 00:01 2024-04-04T20:42:00.437 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:42:00.832 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:42:00.832 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:42:01.401 INFO:teuthology.orchestra.run.smithi106.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-04T20:42:01.401 INFO:teuthology.orchestra.run.smithi106.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-04T20:42:01.486 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:42:01.498 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:42:01.498 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-04T20:42:01.498 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:42:01.498 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:42:01.498 INFO:teuthology.orchestra.run.smithi106.stdout: kernel x86_64 5.14.0-435.el9 CentOS-BaseOS 212 k 2024-04-04T20:42:01.498 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-04T20:42:01.499 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-core x86_64 5.14.0-435.el9 CentOS-BaseOS 15 M 2024-04-04T20:42:01.499 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules x86_64 5.14.0-435.el9 CentOS-BaseOS 34 M 2024-04-04T20:42:01.499 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules-core x86_64 5.14.0-435.el9 CentOS-BaseOS 28 M 2024-04-04T20:42:01.499 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:42:01.499 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:42:01.499 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:42:01.499 INFO:teuthology.orchestra.run.smithi106.stdout:Install 4 Packages 2024-04-04T20:42:01.499 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:42:01.499 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 77 M 2024-04-04T20:42:01.499 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 123 M 2024-04-04T20:42:01.499 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:42:01.654 INFO:teuthology.orchestra.run.smithi106.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 1.3 MB/s | 212 kB 00:00 2024-04-04T20:42:01.679 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:42:01.680 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:42:02.521 INFO:teuthology.orchestra.run.smithi106.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 15 MB/s | 15 MB 00:01 2024-04-04T20:42:03.537 INFO:teuthology.orchestra.run.smithi106.stdout:(3/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 16 MB/s | 34 MB 00:02 2024-04-04T20:42:03.879 INFO:teuthology.orchestra.run.smithi106.stdout:(4/4): kernel-modules-core-5.14.0-435.el9.x86_6 12 MB/s | 28 MB 00:02 2024-04-04T20:42:03.882 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:42:03.882 INFO:teuthology.orchestra.run.smithi106.stdout:Total 32 MB/s | 77 MB 00:02 2024-04-04T20:42:03.883 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:42:03.991 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:42:04.258 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:42:04.259 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:42:04.618 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-04T20:42:04.704 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-04T20:42:05.110 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:42:05.110 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:42:07.393 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:42:07.448 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-04T20:42:07.583 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-04T20:42:07.991 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-04T20:42:08.072 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-04T20:42:10.846 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-04T20:42:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-04T20:42:12.319 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-04T20:42:12.466 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-04T20:42:15.716 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-04T20:42:15.816 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-04T20:42:17.244 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-04T20:42:20.534 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-04T20:42:47.044 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-04T20:42:47.044 INFO:teuthology.orchestra.run.smithi203.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-04T20:42:47.045 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:42:47.087 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-04T20:42:47.469 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-04T20:42:47.470 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-04T20:42:47.470 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-04T20:42:47.470 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-04T20:42:47.878 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-04T20:42:47.879 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:42:47.879 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:42:47.879 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-04T20:42:47.879 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-04T20:42:47.879 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-04T20:42:47.879 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-04T20:42:47.879 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:42:47.879 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:42:48.211 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-04-04T20:42:48.307 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-04T20:42:48.308 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-04T20:42:48.308 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-04T20:42:48.308 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-04T20:42:48.308 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-04T20:42:48.308 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-04T20:42:48.308 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-04-04T20:42:48.309 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-04-04T20:42:48.389 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-04T20:42:48.389 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-04T20:42:48.389 DEBUG:teuthology.orchestra.run.smithi203:> sudo rpm -qi grub2-tools 2024-04-04T20:42:48.423 INFO:teuthology.orchestra.run.smithi203.stdout:Name : grub2-tools 2024-04-04T20:42:48.423 INFO:teuthology.orchestra.run.smithi203.stdout:Epoch : 1 2024-04-04T20:42:48.423 INFO:teuthology.orchestra.run.smithi203.stdout:Version : 2.06 2024-04-04T20:42:48.423 INFO:teuthology.orchestra.run.smithi203.stdout:Release : 68.el9 2024-04-04T20:42:48.423 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: x86_64 2024-04-04T20:42:48.423 INFO:teuthology.orchestra.run.smithi203.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-04T20:42:48.423 INFO:teuthology.orchestra.run.smithi203.stdout:Group : Unspecified 2024-04-04T20:42:48.424 INFO:teuthology.orchestra.run.smithi203.stdout:Size : 8264254 2024-04-04T20:42:48.424 INFO:teuthology.orchestra.run.smithi203.stdout:License : GPLv3+ 2024-04-04T20:42:48.424 INFO:teuthology.orchestra.run.smithi203.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-04T20:42:48.424 INFO:teuthology.orchestra.run.smithi203.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-04T20:42:48.424 INFO:teuthology.orchestra.run.smithi203.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-04T20:42:48.424 INFO:teuthology.orchestra.run.smithi203.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-04T20:42:48.424 INFO:teuthology.orchestra.run.smithi203.stdout:Packager : builder@centos.org 2024-04-04T20:42:48.424 INFO:teuthology.orchestra.run.smithi203.stdout:Vendor : CentOS 2024-04-04T20:42:48.424 INFO:teuthology.orchestra.run.smithi203.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-04T20:42:48.424 INFO:teuthology.orchestra.run.smithi203.stdout:Summary : Support tools for GRUB. 2024-04-04T20:42:48.424 INFO:teuthology.orchestra.run.smithi203.stdout:Description : 2024-04-04T20:42:48.424 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:42:48.424 INFO:teuthology.orchestra.run.smithi203.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-04T20:42:48.424 INFO:teuthology.orchestra.run.smithi203.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-04T20:42:48.425 INFO:teuthology.orchestra.run.smithi203.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-04T20:42:48.425 INFO:teuthology.orchestra.run.smithi203.stdout:hardware devices. 2024-04-04T20:42:48.425 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:42:48.425 INFO:teuthology.orchestra.run.smithi203.stdout:This subpackage provides tools for support of all platforms. 2024-04-04T20:42:48.426 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-04T20:42:48.426 INFO:teuthology.task.kernel:Updating grub on smithi203 to boot 5.14.0-435.el9.x86_64 2024-04-04T20:42:48.426 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-04T20:42:49.037 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-04-04T20:42:49.744 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-04T20:42:49.744 INFO:teuthology.orchestra.run.smithi106.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-04T20:42:49.744 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:42:49.787 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-04T20:42:50.218 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-04T20:42:50.218 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-04T20:42:50.218 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-04T20:42:50.218 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-04T20:42:50.455 INFO:teuthology.orchestra.run.smithi203.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-04T20:42:50.474 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-04-04T20:42:50.476 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-04-04T20:42:50.491 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.ZW4t3hN5S1 2024-04-04T20:42:50.492 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZW4t3hN5S1 2024-04-04T20:42:50.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.ZW4t3hN5S1 2024-04-04T20:42:50.627 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-04T20:42:50.627 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:42:50.627 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:42:50.627 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-04T20:42:50.628 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-04T20:42:50.628 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-04T20:42:50.628 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-04T20:42:50.628 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:42:50.628 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:42:50.696 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.ZW4t3hN5S1 is 6KB 2024-04-04T20:42:50.749 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.ZW4t3hN5S1 2024-04-04T20:42:50.763 DEBUG:teuthology.orchestra.run.smithi203:> sudo /bin/ls /boot/loader/entries || true 2024-04-04T20:42:50.829 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-04T20:42:50.829 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-04T20:42:50.830 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-04T20:42:50.830 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-04T20:42:50.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-04T20:42:50.962 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q kernel | sort -rV | head -n 1 2024-04-04T20:42:50.968 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-04-04T20:42:51.112 INFO:teuthology.orchestra.run.smithi106.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-04T20:42:51.113 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-04T20:42:51.113 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-04T20:42:51.113 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-04T20:42:51.113 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-04T20:42:51.113 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-04T20:42:51.113 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi106.front.sepia.ceph.com, path=None, version=distro) 2024-04-04T20:42:51.113 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q kernel | sort -rV | head -n 1 2024-04-04T20:42:51.193 INFO:teuthology.orchestra.run.smithi106.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-04T20:42:51.194 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-04T20:42:51.194 DEBUG:teuthology.orchestra.run.smithi106:> sudo rpm -qi grub2-tools 2024-04-04T20:42:51.237 INFO:teuthology.orchestra.run.smithi106.stdout:Name : grub2-tools 2024-04-04T20:42:51.237 INFO:teuthology.orchestra.run.smithi106.stdout:Epoch : 1 2024-04-04T20:42:51.237 INFO:teuthology.orchestra.run.smithi106.stdout:Version : 2.06 2024-04-04T20:42:51.237 INFO:teuthology.orchestra.run.smithi106.stdout:Release : 68.el9 2024-04-04T20:42:51.237 INFO:teuthology.orchestra.run.smithi106.stdout:Architecture: x86_64 2024-04-04T20:42:51.237 INFO:teuthology.orchestra.run.smithi106.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-04T20:42:51.237 INFO:teuthology.orchestra.run.smithi106.stdout:Group : Unspecified 2024-04-04T20:42:51.237 INFO:teuthology.orchestra.run.smithi106.stdout:Size : 8264254 2024-04-04T20:42:51.237 INFO:teuthology.orchestra.run.smithi106.stdout:License : GPLv3+ 2024-04-04T20:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-04T20:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-04T20:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-04T20:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-04T20:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout:Packager : builder@centos.org 2024-04-04T20:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout:Vendor : CentOS 2024-04-04T20:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-04T20:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout:Summary : Support tools for GRUB. 2024-04-04T20:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout:Description : 2024-04-04T20:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-04T20:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-04T20:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-04T20:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout:hardware devices. 2024-04-04T20:42:51.239 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:42:51.239 INFO:teuthology.orchestra.run.smithi106.stdout:This subpackage provides tools for support of all platforms. 2024-04-04T20:42:51.239 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-04T20:42:51.240 INFO:teuthology.task.kernel:Updating grub on smithi106 to boot 5.14.0-435.el9.x86_64 2024-04-04T20:42:51.240 DEBUG:teuthology.orchestra.run.smithi106:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-04T20:42:51.883 INFO:teuthology.orchestra.run.smithi106.stderr:Generating grub configuration file ... 2024-04-04T20:42:53.145 INFO:teuthology.orchestra.run.smithi106.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-04T20:42:53.182 INFO:teuthology.orchestra.run.smithi106.stderr:done 2024-04-04T20:42:53.184 DEBUG:teuthology.orchestra.run.smithi106:> mktemp 2024-04-04T20:42:53.199 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.31zq7EXwdn 2024-04-04T20:42:53.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.31zq7EXwdn 2024-04-04T20:42:53.275 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.31zq7EXwdn 2024-04-04T20:42:53.415 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.31zq7EXwdn is 6KB 2024-04-04T20:42:53.464 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.31zq7EXwdn 2024-04-04T20:42:53.479 DEBUG:teuthology.orchestra.run.smithi106:> sudo /bin/ls /boot/loader/entries || true 2024-04-04T20:42:53.544 INFO:teuthology.orchestra.run.smithi106.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-04T20:42:53.545 INFO:teuthology.orchestra.run.smithi106.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-04T20:42:53.545 INFO:teuthology.orchestra.run.smithi106.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-04T20:42:53.545 INFO:teuthology.orchestra.run.smithi106.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-04T20:42:53.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-04T20:42:53.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo shutdown -r now 2024-04-04T20:43:20.998 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-04T20:43:20.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-04T20:43:21.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-04T20:43:23.703 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-04T20:43:23.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-04-04T20:43:23.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-04T20:43:55.551 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-04-04T20:44:04.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-04T20:44:04.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-04T20:44:23.722 DEBUG:teuthology.orchestra.remote:timed out 2024-04-04T20:44:32.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-04-04T20:44:32.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-04T20:44:33.115 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-04-04T20:44:33.536 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-04-04T20:44:33.536 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-435.el9.x86_64"... 2024-04-04T20:44:33.536 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2024-04-04T20:44:33.551 INFO:teuthology.orchestra.run.smithi106.stdout:5.14.0-435.el9.x86_64 2024-04-04T20:44:33.551 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-04T20:44:33.551 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-04T20:44:33.551 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-04T20:44:34.552 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-04T20:44:34.552 DEBUG:teuthology.orchestra.run.smithi106:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-04T20:44:34.632 INFO:teuthology.orchestra.run.smithi106.stdout:ttyS1 2024-04-04T20:44:34.674 DEBUG:teuthology.parallel:result is None 2024-04-04T20:45:04.562 DEBUG:teuthology.orchestra.remote:timed out 2024-04-04T20:45:16.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-04T20:45:16.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-04T20:45:16.837 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-04-04T20:45:17.226 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-04T20:45:17.226 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-435.el9.x86_64"... 2024-04-04T20:45:17.226 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-04-04T20:45:17.241 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-435.el9.x86_64 2024-04-04T20:45:17.241 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-04T20:45:17.241 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-04T20:45:17.242 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-04T20:45:18.242 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-04T20:45:18.243 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-04T20:45:18.326 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-04-04T20:45:18.359 DEBUG:teuthology.parallel:result is None 2024-04-04T20:45:18.359 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-04T20:45:18.366 INFO:teuthology.task.internal:Creating test directory... 2024-04-04T20:45:18.366 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-04T20:45:18.370 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-04T20:45:18.387 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-04T20:45:18.393 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-04T20:45:18.400 INFO:teuthology.task.internal:Creating archive directory... 2024-04-04T20:45:18.401 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-04T20:45:18.427 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-04T20:45:18.459 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-04T20:45:18.466 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-04T20:45:18.466 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-04T20:45:18.488 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-04T20:45:18.526 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-04T20:45:18.531 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-04T20:45:18.540 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-04T20:45:18.545 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-04T20:45:18.546 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-04T20:45:18.553 INFO:teuthology.task.internal:Configuring sudo... 2024-04-04T20:45:18.553 DEBUG:teuthology.orchestra.run.smithi106:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-04T20:45:18.584 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-04T20:45:18.616 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-04T20:45:18.626 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-04T20:45:18.626 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-04T20:45:18.654 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-04T20:45:18.672 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-04T20:45:18.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-04T20:45:18.795 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-04T20:45:18.875 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-04T20:45:18.919 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T20:45:18.919 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-04T20:45:18.985 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-04T20:45:19.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-04T20:45:19.095 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-04T20:45:19.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-04T20:45:19.183 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T20:45:19.183 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-04T20:45:19.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo service rsyslog restart 2024-04-04T20:45:19.255 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-04-04T20:45:19.299 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-04T20:45:19.337 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-04T20:45:19.794 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-04T20:45:19.802 INFO:teuthology.task.internal:Starting timer... 2024-04-04T20:45:19.803 INFO:teuthology.run_tasks:Running task pcp... 2024-04-04T20:45:19.813 INFO:teuthology.run_tasks:Running task selinux... 2024-04-04T20:45:19.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo service auditd rotate 2024-04-04T20:45:19.893 INFO:teuthology.orchestra.run.smithi106.stdout:Rotating logs: 2024-04-04T20:45:19.895 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-04-04T20:45:19.951 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-04-04T20:45:19.953 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-04T20:45:19.953 DEBUG:teuthology.orchestra.run.smithi106:> /usr/sbin/getenforce 2024-04-04T20:45:19.975 INFO:teuthology.orchestra.run.smithi106.stdout:Permissive 2024-04-04T20:45:19.975 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-04-04T20:45:20.017 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-04-04T20:45:20.017 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi106.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-04-04T20:45:20.017 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-04T20:45:20.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-04T20:45:20.046 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-04T20:45:20.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-04T20:45:20.086 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-04T20:45:20.087 DEBUG:teuthology.orchestra.run.smithi106:> sudo /usr/sbin/setenforce permissive 2024-04-04T20:45:20.119 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-04-04T20:45:20.160 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-04T20:45:20.169 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-04T20:45:20.174 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-04T20:45:20.359 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-04T20:45:20.375 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-04T20:45:20.377 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi106.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-04T20:49:31.365 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi106.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-04-04T20:49:31.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-04-04T20:49:31.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-04T20:49:31.460 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-04-04T20:49:31.537 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-04-04T20:49:31.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-04T20:49:31.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-04T20:49:31.626 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-04-04T20:49:31.697 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-04T20:49:31.698 INFO:teuthology.run_tasks:Running task clock... 2024-04-04T20:49:31.709 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-04T20:49:31.709 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-04T20:49:31.709 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-04T20:49:31.712 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-04T20:49:31.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-04T20:49:31.742 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-04T20:49:31.759 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-04T20:49:31.768 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-04T20:49:31.784 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-04T20:49:31.789 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: ntpd: command not found 2024-04-04T20:49:31.803 INFO:teuthology.orchestra.run.smithi106.stdout:506 Cannot talk to daemon 2024-04-04T20:49:31.813 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-04-04T20:49:31.820 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-04T20:49:31.827 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-04-04T20:49:31.836 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-04T20:49:31.844 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-04T20:49:31.860 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-04T20:49:31.889 INFO:teuthology.orchestra.run.smithi106.stderr:bash: line 1: ntpq: command not found 2024-04-04T20:49:31.892 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-04T20:49:31.892 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-04-04T20:49:31.892 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-04T20:49:31.892 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-04T20:49:31.893 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-04T20:49:31.893 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-04T20:49:31.921 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-04-04T20:49:31.925 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-04T20:49:31.926 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-04-04T20:49:31.926 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-04T20:49:31.926 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-04T20:49:31.926 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-04T20:49:31.926 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-04T20:49:31.926 INFO:teuthology.run_tasks:Running task install... 2024-04-04T20:49:31.935 DEBUG:teuthology.task.install:project ceph 2024-04-04T20:49:31.935 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '66deef1598934c3b1ef2f044e380a7d00244cead'}} 2024-04-04T20:49:31.935 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '66deef1598934c3b1ef2f044e380a7d00244cead'} 2024-04-04T20:49:31.935 INFO:teuthology.task.install:Using flavor: default 2024-04-04T20:49:31.941 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-04-04T20:49:31.941 INFO:teuthology.task.install:extra packages: [] 2024-04-04T20:49:31.941 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '66deef1598934c3b1ef2f044e380a7d00244cead', 'tag': None, 'wait_for_package': False} 2024-04-04T20:49:31.942 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=66deef1598934c3b1ef2f044e380a7d00244cead 2024-04-04T20:49:31.944 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '66deef1598934c3b1ef2f044e380a7d00244cead', 'tag': None, 'wait_for_package': False} 2024-04-04T20:49:31.944 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=66deef1598934c3b1ef2f044e380a7d00244cead 2024-04-04T20:49:32.091 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/66deef1598934c3b1ef2f044e380a7d00244cead/centos/9/flavors/default/ 2024-04-04T20:49:32.091 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1864.g66deef15 2024-04-04T20:49:32.264 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/66deef1598934c3b1ef2f044e380a7d00244cead/centos/9/flavors/default/ 2024-04-04T20:49:32.265 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1864.g66deef15 2024-04-04T20:49:32.367 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/66deef1598934c3b1ef2f044e380a7d00244cead/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/66deef1598934c3b1ef2f044e380a7d00244cead/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/66deef1598934c3b1ef2f044e380a7d00244cead/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-04T20:49:32.367 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T20:49:32.367 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-04T20:49:32.396 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-04-04T20:49:32.396 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/66deef1598934c3b1ef2f044e380a7d00244cead/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-04T20:49:32.467 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-04T20:49:32.473 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/66deef1598934c3b1ef2f044e380a7d00244cead/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/66deef1598934c3b1ef2f044e380a7d00244cead/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/66deef1598934c3b1ef2f044e380a7d00244cead/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-04T20:49:32.473 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T20:49:32.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-04T20:49:32.503 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-04-04T20:49:32.503 DEBUG:teuthology.orchestra.run.smithi106:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/66deef1598934c3b1ef2f044e380a7d00244cead/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-04T20:49:32.547 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-04T20:49:32.572 DEBUG:teuthology.orchestra.run.smithi106:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-04T20:49:32.613 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-04-04T20:49:32.614 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-04-04T20:49:32.653 DEBUG:teuthology.orchestra.run.smithi106:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-04T20:49:32.720 INFO:teuthology.orchestra.run.smithi106.stdout:check_obsoletes = 1 2024-04-04T20:49:32.722 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum clean all 2024-04-04T20:49:32.851 INFO:teuthology.orchestra.run.smithi203.stdout:74 files removed 2024-04-04T20:49:32.877 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw 2024-04-04T20:49:32.965 INFO:teuthology.orchestra.run.smithi106.stdout:74 files removed 2024-04-04T20:49:32.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-radosgw 2024-04-04T20:49:33.587 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 202 kB/s | 82 kB 00:00 2024-04-04T20:49:33.755 INFO:teuthology.orchestra.run.smithi106.stdout:ceph packages for x86_64 177 kB/s | 82 kB 00:00 2024-04-04T20:49:34.004 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-04-04T20:49:34.114 INFO:teuthology.orchestra.run.smithi106.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-04-04T20:49:34.354 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-04-04T20:49:34.423 INFO:teuthology.orchestra.run.smithi106.stdout:ceph source packages 7.0 kB/s | 1.9 kB 00:00 2024-04-04T20:49:34.790 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-04-04T20:49:34.892 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-04-04T20:49:36.240 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-04-04T20:49:36.293 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-04-04T20:49:38.117 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.0 MB 00:01 2024-04-04T20:49:39.876 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-04-04T20:49:46.119 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.1 MB 00:02 2024-04-04T20:49:46.263 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 848 kB/s | 8.0 MB 00:09 2024-04-04T20:49:47.653 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 16 kB 00:00 2024-04-04T20:49:50.797 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 19 MB 00:03 2024-04-04T20:49:51.029 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 6.4 MB/s | 21 MB 00:03 2024-04-04T20:49:55.182 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 6.1 MB 00:00 2024-04-04T20:49:55.995 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-04T20:49:56.549 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 82 kB/s | 16 kB 00:00 2024-04-04T20:49:58.616 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:49:58.635 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:49:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-04T20:49:58.636 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:49:58.636 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:49:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw x86_64 2:19.0.0-1864.g66deef15.el9 ceph 9.8 M 2024-04-04T20:49:58.636 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrading: 2024-04-04T20:49:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: librados2 x86_64 2:19.0.0-1864.g66deef15.el9 ceph 3.4 M 2024-04-04T20:49:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1 x86_64 2:19.0.0-1864.g66deef15.el9 ceph 3.1 M 2024-04-04T20:49:58.636 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-04T20:49:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: boost-program-options 2024-04-04T20:49:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-04T20:49:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base x86_64 2:19.0.0-1864.g66deef15.el9 ceph 5.3 M 2024-04-04T20:49:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common x86_64 2:19.0.0-1864.g66deef15.el9 ceph 20 M 2024-04-04T20:49:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux x86_64 2:19.0.0-1864.g66deef15.el9 ceph 25 k 2024-04-04T20:49:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-04T20:49:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-04T20:49:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-04T20:49:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2 x86_64 2:19.0.0-1864.g66deef15.el9 ceph 744 k 2024-04-04T20:49:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-04T20:49:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-04T20:49:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-04T20:49:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1 x86_64 2:19.0.0-1864.g66deef15.el9 ceph 501 k 2024-04-04T20:49:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-04T20:49:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2 x86_64 2:19.0.0-1864.g66deef15.el9 ceph 5.1 M 2024-04-04T20:49:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-04T20:49:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-04T20:49:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-04T20:49:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse 2024-04-04T20:49:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: x86_64 2:19.0.0-1864.g66deef15.el9 ceph 45 k 2024-04-04T20:49:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-common x86_64 2:19.0.0-1864.g66deef15.el9 ceph 131 k 2024-04-04T20:49:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs x86_64 2:19.0.0-1864.g66deef15.el9 ceph 162 k 2024-04-04T20:49:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-04T20:49:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados x86_64 2:19.0.0-1864.g66deef15.el9 ceph 320 k 2024-04-04T20:49:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd x86_64 2:19.0.0-1864.g66deef15.el9 ceph 298 k 2024-04-04T20:49:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw x86_64 2:19.0.0-1864.g66deef15.el9 ceph 99 k 2024-04-04T20:49:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-04T20:49:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-04T20:49:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-04T20:49:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-04T20:49:58.639 INFO:teuthology.orchestra.run.smithi106.stdout:Installing weak dependencies: 2024-04-04T20:49:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-04T20:49:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:49:58.639 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:49:58.639 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:49:58.639 INFO:teuthology.orchestra.run.smithi106.stdout:Install 30 Packages 2024-04-04T20:49:58.639 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrade 2 Packages 2024-04-04T20:49:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:49:58.640 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 58 M 2024-04-04T20:49:58.640 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:49:59.572 INFO:teuthology.orchestra.run.smithi106.stdout:(1/32): ceph-base-19.0.0-1864.g66deef15.el9.x86 6.8 MB/s | 5.3 MB 00:00 2024-04-04T20:49:59.613 INFO:teuthology.orchestra.run.smithi106.stdout:(2/32): ceph-selinux-19.0.0-1864.g66deef15.el9. 608 kB/s | 25 kB 00:00 2024-04-04T20:49:59.714 INFO:teuthology.orchestra.run.smithi106.stdout:(3/32): libcephfs2-19.0.0-1864.g66deef15.el9.x8 7.3 MB/s | 744 kB 00:00 2024-04-04T20:49:59.814 INFO:teuthology.orchestra.run.smithi106.stdout:(4/32): libradosstriper1-19.0.0-1864.g66deef15. 4.9 MB/s | 501 kB 00:00 2024-04-04T20:49:59.931 INFO:teuthology.orchestra.run.smithi106.stdout:(5/32): ceph-radosgw-19.0.0-1864.g66deef15.el9. 8.5 MB/s | 9.8 MB 00:01 2024-04-04T20:49:59.990 INFO:teuthology.orchestra.run.smithi106.stdout:(6/32): python3-ceph-argparse-19.0.0-1864.g66de 779 kB/s | 45 kB 00:00 2024-04-04T20:50:00.048 INFO:teuthology.orchestra.run.smithi106.stdout:(7/32): python3-ceph-common-19.0.0-1864.g66deef 2.2 MB/s | 131 kB 00:00 2024-04-04T20:50:00.115 INFO:teuthology.orchestra.run.smithi106.stdout:(8/32): python3-cephfs-19.0.0-1864.g66deef15.el 2.4 MB/s | 162 kB 00:00 2024-04-04T20:50:00.190 INFO:teuthology.orchestra.run.smithi106.stdout:(9/32): python3-rados-19.0.0-1864.g66deef15.el9 4.2 MB/s | 320 kB 00:00 2024-04-04T20:50:00.257 INFO:teuthology.orchestra.run.smithi106.stdout:(10/32): librgw2-19.0.0-1864.g66deef15.el9.x86_ 12 MB/s | 5.1 MB 00:00 2024-04-04T20:50:00.282 INFO:teuthology.orchestra.run.smithi106.stdout:(11/32): python3-rbd-19.0.0-1864.g66deef15.el9. 3.2 MB/s | 298 kB 00:00 2024-04-04T20:50:00.317 INFO:teuthology.orchestra.run.smithi106.stdout:(12/32): python3-rgw-19.0.0-1864.g66deef15.el9. 1.7 MB/s | 99 kB 00:00 2024-04-04T20:50:00.433 INFO:teuthology.orchestra.run.smithi106.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 704 kB/s | 106 kB 00:00 2024-04-04T20:50:00.458 INFO:teuthology.orchestra.run.smithi106.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-04-04T20:50:00.493 INFO:teuthology.orchestra.run.smithi106.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 748 kB/s | 44 kB 00:00 2024-04-04T20:50:00.551 INFO:teuthology.orchestra.run.smithi106.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-04-04T20:50:00.558 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 5.4 MB/s | 21 MB 00:03 2024-04-04T20:50:00.577 INFO:teuthology.orchestra.run.smithi106.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-04-04T20:50:00.601 INFO:teuthology.orchestra.run.smithi106.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 826 kB/s | 41 kB 00:00 2024-04-04T20:50:00.627 INFO:teuthology.orchestra.run.smithi106.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-04-04T20:50:00.652 INFO:teuthology.orchestra.run.smithi106.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 696 kB/s | 35 kB 00:00 2024-04-04T20:50:00.677 INFO:teuthology.orchestra.run.smithi106.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 923 kB/s | 46 kB 00:00 2024-04-04T20:50:00.961 INFO:teuthology.orchestra.run.smithi106.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-04-04T20:50:01.045 INFO:teuthology.orchestra.run.smithi106.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 313 kB/s | 26 kB 00:00 2024-04-04T20:50:01.170 INFO:teuthology.orchestra.run.smithi106.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 389 kB/s | 49 kB 00:00 2024-04-04T20:50:01.362 INFO:teuthology.orchestra.run.smithi106.stdout:(25/32): ceph-common-19.0.0-1864.g66deef15.el9. 7.7 MB/s | 20 MB 00:02 2024-04-04T20:50:01.404 INFO:teuthology.orchestra.run.smithi106.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 289 kB/s | 67 kB 00:00 2024-04-04T20:50:01.596 INFO:teuthology.orchestra.run.smithi106.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-04-04T20:50:01.621 INFO:teuthology.orchestra.run.smithi106.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.2 MB/s | 838 kB 00:00 2024-04-04T20:50:01.771 INFO:teuthology.orchestra.run.smithi106.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.0 MB/s | 4.4 MB 00:01 2024-04-04T20:50:01.872 INFO:teuthology.orchestra.run.smithi106.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.8 MB/s | 1.6 MB 00:00 2024-04-04T20:50:01.980 INFO:teuthology.orchestra.run.smithi106.stdout:(31/32): librados2-19.0.0-1864.g66deef15.el9.x8 9.4 MB/s | 3.4 MB 00:00 2024-04-04T20:50:02.064 INFO:teuthology.orchestra.run.smithi106.stdout:(32/32): librbd1-19.0.0-1864.g66deef15.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-04-04T20:50:02.069 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:50:02.069 INFO:teuthology.orchestra.run.smithi106.stdout:Total 17 MB/s | 58 MB 00:03 2024-04-04T20:50:02.069 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:50:02.104 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:50:02.104 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:50:02.586 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:50:02.587 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:50:04.045 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:50:04.129 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-04T20:50:04.168 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-04T20:50:04.194 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-04T20:50:04.217 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-04T20:50:04.234 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-04T20:50:04.394 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-04T20:50:04.448 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-04T20:50:04.789 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-04T20:50:04.854 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librados2-2:19.0.0-1864.g66deef15.el9.x86_64 7/34 2024-04-04T20:50:04.946 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-2:19.0.0-1864.g66deef15.el9.x86_64 7/34 2024-04-04T20:50:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rados-2:19.0.0-1864.g66deef15.el9.x86_64 8/34 2024-04-04T20:50:05.072 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs2-2:19.0.0-1864.g66deef15.el9.x86_64 9/34 2024-04-04T20:50:05.136 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephfs2-2:19.0.0-1864.g66deef15.el9.x86_64 9/34 2024-04-04T20:50:05.176 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-04T20:50:05.216 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-argparse-2:19.0.0-1864.g66deef15.el9. 11/34 2024-04-04T20:50:05.290 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cephfs-2:19.0.0-1864.g66deef15.el9.x86_64 12/34 2024-04-04T20:50:05.316 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libradosstriper1-2:19.0.0-1864.g66deef15.el9.x86_6 13/34 2024-04-04T20:50:05.371 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1864.g66deef15.el9.x86_6 13/34 2024-04-04T20:50:05.406 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-04T20:50:05.513 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-04T20:50:05.541 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-04T20:50:05.581 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-04T20:50:05.600 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-04T20:50:05.615 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-04T20:50:05.651 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-04T20:50:05.684 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-04T20:50:05.996 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-04T20:50:06.022 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librbd1-2:19.0.0-1864.g66deef15.el9.x86_64 22/34 2024-04-04T20:50:06.112 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-2:19.0.0-1864.g66deef15.el9.x86_64 22/34 2024-04-04T20:50:06.143 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rbd-2:19.0.0-1864.g66deef15.el9.x86_64 23/34 2024-04-04T20:50:06.185 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-04T20:50:06.632 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-04T20:50:06.658 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librgw2-2:19.0.0-1864.g66deef15.el9.x86_64 26/34 2024-04-04T20:50:06.713 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librgw2-2:19.0.0-1864.g66deef15.el9.x86_64 26/34 2024-04-04T20:50:06.781 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rgw-2:19.0.0-1864.g66deef15.el9.x86_64 27/34 2024-04-04T20:50:06.818 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-common-2:19.0.0-1864.g66deef15.el9.x8 28/34 2024-04-04T20:50:08.211 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:50:08.225 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:50:08.225 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-04T20:50:08.225 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:50:08.225 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:50:08.226 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.0.0-1864.g66deef15.el9 ceph 9.8 M 2024-04-04T20:50:08.226 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-04-04T20:50:08.226 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.0.0-1864.g66deef15.el9 ceph 3.4 M 2024-04-04T20:50:08.226 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.0.0-1864.g66deef15.el9 ceph 3.1 M 2024-04-04T20:50:08.226 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-04T20:50:08.226 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options 2024-04-04T20:50:08.226 INFO:teuthology.orchestra.run.smithi203.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-04T20:50:08.226 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.0.0-1864.g66deef15.el9 ceph 5.3 M 2024-04-04T20:50:08.226 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.0.0-1864.g66deef15.el9 ceph 20 M 2024-04-04T20:50:08.226 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.0.0-1864.g66deef15.el9 ceph 25 k 2024-04-04T20:50:08.226 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-04T20:50:08.226 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-04T20:50:08.227 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-04T20:50:08.227 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.0.0-1864.g66deef15.el9 ceph 744 k 2024-04-04T20:50:08.227 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-04T20:50:08.227 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-04T20:50:08.227 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-04T20:50:08.227 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.0.0-1864.g66deef15.el9 ceph 501 k 2024-04-04T20:50:08.227 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-04T20:50:08.227 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.0.0-1864.g66deef15.el9 ceph 5.1 M 2024-04-04T20:50:08.227 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-04T20:50:08.227 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-04T20:50:08.227 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-04T20:50:08.227 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse 2024-04-04T20:50:08.227 INFO:teuthology.orchestra.run.smithi203.stdout: x86_64 2:19.0.0-1864.g66deef15.el9 ceph 45 k 2024-04-04T20:50:08.227 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.0.0-1864.g66deef15.el9 ceph 131 k 2024-04-04T20:50:08.228 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.0.0-1864.g66deef15.el9 ceph 162 k 2024-04-04T20:50:08.228 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-04T20:50:08.228 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.0.0-1864.g66deef15.el9 ceph 320 k 2024-04-04T20:50:08.228 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.0.0-1864.g66deef15.el9 ceph 298 k 2024-04-04T20:50:08.228 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.0.0-1864.g66deef15.el9 ceph 99 k 2024-04-04T20:50:08.228 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-04T20:50:08.228 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-04T20:50:08.228 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-04T20:50:08.228 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-04T20:50:08.228 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-04-04T20:50:08.228 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-04T20:50:08.228 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:08.228 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:50:08.228 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:50:08.229 INFO:teuthology.orchestra.run.smithi203.stdout:Install 30 Packages 2024-04-04T20:50:08.229 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 2 Packages 2024-04-04T20:50:08.229 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:08.229 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 58 M 2024-04-04T20:50:08.229 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:50:08.964 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:19.0.0-1864.g66deef15.el9.x86_64 29/34 2024-04-04T20:50:08.998 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-common-2:19.0.0-1864.g66deef15.el9.x86_64 29/34 2024-04-04T20:50:09.043 INFO:teuthology.orchestra.run.smithi203.stdout:(1/32): ceph-base-19.0.0-1864.g66deef15.el9.x86 7.3 MB/s | 5.3 MB 00:00 2024-04-04T20:50:09.135 INFO:teuthology.orchestra.run.smithi203.stdout:(2/32): ceph-selinux-19.0.0-1864.g66deef15.el9. 275 kB/s | 25 kB 00:00 2024-04-04T20:50:09.252 INFO:teuthology.orchestra.run.smithi203.stdout:(3/32): libcephfs2-19.0.0-1864.g66deef15.el9.x8 6.2 MB/s | 744 kB 00:00 2024-04-04T20:50:09.352 INFO:teuthology.orchestra.run.smithi203.stdout:(4/32): libradosstriper1-19.0.0-1864.g66deef15. 4.9 MB/s | 501 kB 00:00 2024-04-04T20:50:09.553 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:19.0.0-1864.g66deef15.el9.x86_64 29/34 2024-04-04T20:50:09.593 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-base-2:19.0.0-1864.g66deef15.el9.x86_64 30/34 2024-04-04T20:50:09.687 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-base-2:19.0.0-1864.g66deef15.el9.x86_64 30/34 2024-04-04T20:50:09.687 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-04T20:50:09.687 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-04T20:50:09.687 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:09.720 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-selinux-2:19.0.0-1864.g66deef15.el9.x86_64 31/34 2024-04-04T20:50:09.835 INFO:teuthology.orchestra.run.smithi203.stdout:(5/32): librgw2-19.0.0-1864.g66deef15.el9.x86_6 11 MB/s | 5.1 MB 00:00 2024-04-04T20:50:09.944 INFO:teuthology.orchestra.run.smithi203.stdout:(6/32): ceph-radosgw-19.0.0-1864.g66deef15.el9. 6.0 MB/s | 9.8 MB 00:01 2024-04-04T20:50:09.961 INFO:teuthology.orchestra.run.smithi203.stdout:(7/32): python3-ceph-argparse-19.0.0-1864.g66de 359 kB/s | 45 kB 00:00 2024-04-04T20:50:10.003 INFO:teuthology.orchestra.run.smithi203.stdout:(8/32): python3-ceph-common-19.0.0-1864.g66deef 2.2 MB/s | 131 kB 00:00 2024-04-04T20:50:10.028 INFO:teuthology.orchestra.run.smithi203.stdout:(9/32): python3-cephfs-19.0.0-1864.g66deef15.el 2.4 MB/s | 162 kB 00:00 2024-04-04T20:50:10.071 INFO:teuthology.orchestra.run.smithi203.stdout:(10/32): python3-rados-19.0.0-1864.g66deef15.el 4.7 MB/s | 320 kB 00:00 2024-04-04T20:50:10.096 INFO:teuthology.orchestra.run.smithi203.stdout:(11/32): python3-rbd-19.0.0-1864.g66deef15.el9. 4.3 MB/s | 298 kB 00:00 2024-04-04T20:50:10.129 INFO:teuthology.orchestra.run.smithi203.stdout:(12/32): python3-rgw-19.0.0-1864.g66deef15.el9. 1.7 MB/s | 99 kB 00:00 2024-04-04T20:50:10.363 INFO:teuthology.orchestra.run.smithi203.stdout:(13/32): ceph-common-19.0.0-1864.g66deef15.el9. 9.7 MB/s | 20 MB 00:02 2024-04-04T20:50:10.438 INFO:teuthology.orchestra.run.smithi203.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 309 kB/s | 106 kB 00:00 2024-04-04T20:50:10.463 INFO:teuthology.orchestra.run.smithi203.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 617 kB/s | 44 kB 00:00 2024-04-04T20:50:10.497 INFO:teuthology.orchestra.run.smithi203.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 466 kB/s | 159 kB 00:00 2024-04-04T20:50:10.531 INFO:teuthology.orchestra.run.smithi203.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-04-04T20:50:10.556 INFO:teuthology.orchestra.run.smithi203.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-04-04T20:50:10.581 INFO:teuthology.orchestra.run.smithi203.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 497 kB/s | 41 kB 00:00 2024-04-04T20:50:10.606 INFO:teuthology.orchestra.run.smithi203.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-04-04T20:50:10.723 INFO:teuthology.orchestra.run.smithi203.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 209 kB/s | 35 kB 00:00 2024-04-04T20:50:10.748 INFO:teuthology.orchestra.run.smithi203.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 277 kB/s | 46 kB 00:00 2024-04-04T20:50:11.174 INFO:teuthology.orchestra.run.smithi203.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 61 kB/s | 26 kB 00:00 2024-04-04T20:50:11.366 INFO:teuthology.orchestra.run.smithi203.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-04-04T20:50:11.483 INFO:teuthology.orchestra.run.smithi203.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 407 kB/s | 309 kB 00:00 2024-04-04T20:50:11.508 INFO:teuthology.orchestra.run.smithi203.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 475 kB/s | 67 kB 00:00 2024-04-04T20:50:11.741 INFO:teuthology.orchestra.run.smithi203.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 822 kB/s | 192 kB 00:00 2024-04-04T20:50:11.775 INFO:teuthology.orchestra.run.smithi203.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 838 kB 00:00 2024-04-04T20:50:11.842 INFO:teuthology.orchestra.run.smithi203.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.0 MB/s | 4.4 MB 00:01 2024-04-04T20:50:12.076 INFO:teuthology.orchestra.run.smithi203.stdout:(30/32): librados2-19.0.0-1864.g66deef15.el9.x8 11 MB/s | 3.4 MB 00:00 2024-04-04T20:50:12.209 INFO:teuthology.orchestra.run.smithi203.stdout:(31/32): librbd1-19.0.0-1864.g66deef15.el9.x86_ 8.6 MB/s | 3.1 MB 00:00 2024-04-04T20:50:12.242 INFO:teuthology.orchestra.run.smithi203.stdout:(32/32): thrift-0.15.0-2.el9.x86_64.rpm 3.2 MB/s | 1.6 MB 00:00 2024-04-04T20:50:12.249 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:50:12.250 INFO:teuthology.orchestra.run.smithi203.stdout:Total 14 MB/s | 58 MB 00:04 2024-04-04T20:50:12.250 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:50:12.288 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:50:12.288 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:50:12.771 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:50:12.771 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:50:14.251 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:50:14.336 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-04T20:50:14.364 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-04T20:50:14.392 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-04T20:50:14.412 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-04T20:50:14.430 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-04T20:50:14.579 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-04T20:50:14.634 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-04T20:50:14.961 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-04T20:50:15.032 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.0.0-1864.g66deef15.el9.x86_64 7/34 2024-04-04T20:50:15.124 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.0.0-1864.g66deef15.el9.x86_64 7/34 2024-04-04T20:50:15.211 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.0.0-1864.g66deef15.el9.x86_64 8/34 2024-04-04T20:50:15.251 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.0.0-1864.g66deef15.el9.x86_64 9/34 2024-04-04T20:50:15.313 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.0.0-1864.g66deef15.el9.x86_64 9/34 2024-04-04T20:50:15.342 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-04T20:50:15.393 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.0.0-1864.g66deef15.el9. 11/34 2024-04-04T20:50:15.466 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.0.0-1864.g66deef15.el9.x86_64 12/34 2024-04-04T20:50:15.495 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.0.0-1864.g66deef15.el9.x86_6 13/34 2024-04-04T20:50:15.558 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1864.g66deef15.el9.x86_6 13/34 2024-04-04T20:50:15.593 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-04T20:50:15.710 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-04T20:50:15.775 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-04T20:50:15.818 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-04T20:50:15.851 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-04T20:50:15.880 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-04T20:50:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-04T20:50:16.227 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-04T20:50:16.259 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.0.0-1864.g66deef15.el9.x86_64 22/34 2024-04-04T20:50:16.348 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.0.0-1864.g66deef15.el9.x86_64 22/34 2024-04-04T20:50:16.380 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.0.0-1864.g66deef15.el9.x86_64 23/34 2024-04-04T20:50:16.420 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-04T20:50:16.880 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-04T20:50:16.911 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.0.0-1864.g66deef15.el9.x86_64 26/34 2024-04-04T20:50:16.983 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.0.0-1864.g66deef15.el9.x86_64 26/34 2024-04-04T20:50:17.050 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.0.0-1864.g66deef15.el9.x86_64 27/34 2024-04-04T20:50:17.080 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.0.0-1864.g66deef15.el9.x8 28/34 2024-04-04T20:50:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1864.g66deef15.el9.x86_64 31/34 2024-04-04T20:50:19.122 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /sys 2024-04-04T20:50:19.122 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /proc 2024-04-04T20:50:19.122 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /mnt 2024-04-04T20:50:19.122 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /var/tmp 2024-04-04T20:50:19.122 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /home 2024-04-04T20:50:19.122 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /root 2024-04-04T20:50:19.122 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /tmp 2024-04-04T20:50:19.122 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:19.168 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-1864.g66deef15.el9.x86_64 29/34 2024-04-04T20:50:19.218 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.0.0-1864.g66deef15.el9.x86_64 29/34 2024-04-04T20:50:19.780 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-1864.g66deef15.el9.x86_64 29/34 2024-04-04T20:50:19.813 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.0.0-1864.g66deef15.el9.x86_64 30/34 2024-04-04T20:50:19.960 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.0.0-1864.g66deef15.el9.x86_64 30/34 2024-04-04T20:50:19.960 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-04T20:50:19.960 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-04T20:50:19.960 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:19.998 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.0.0-1864.g66deef15.el9.x86_64 31/34 2024-04-04T20:50:20.050 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-radosgw-2:19.0.0-1864.g66deef15.el9.x86_64 32/34 2024-04-04T20:50:20.074 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1864.g66deef15.el9.x86_64 32/34 2024-04-04T20:50:20.075 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:50:20.075 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-04T20:50:20.075 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-04T20:50:20.075 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-04T20:50:20.075 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:20.076 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-04T20:50:20.161 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-04T20:50:20.161 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-04T20:50:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-04T20:50:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-base-2:19.0.0-1864.g66deef15.el9.x86_64 1/34 2024-04-04T20:50:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-common-2:19.0.0-1864.g66deef15.el9.x86_64 2/34 2024-04-04T20:50:21.258 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-radosgw-2:19.0.0-1864.g66deef15.el9.x86_64 3/34 2024-04-04T20:50:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-selinux-2:19.0.0-1864.g66deef15.el9.x86_64 4/34 2024-04-04T20:50:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs2-2:19.0.0-1864.g66deef15.el9.x86_64 5/34 2024-04-04T20:50:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libradosstriper1-2:19.0.0-1864.g66deef15.el9.x86_6 6/34 2024-04-04T20:50:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librgw2-2:19.0.0-1864.g66deef15.el9.x86_64 7/34 2024-04-04T20:50:21.261 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1864.g66deef15.el9. 8/34 2024-04-04T20:50:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-common-2:19.0.0-1864.g66deef15.el9.x8 9/34 2024-04-04T20:50:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cephfs-2:19.0.0-1864.g66deef15.el9.x86_64 10/34 2024-04-04T20:50:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rados-2:19.0.0-1864.g66deef15.el9.x86_64 11/34 2024-04-04T20:50:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rbd-2:19.0.0-1864.g66deef15.el9.x86_64 12/34 2024-04-04T20:50:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rgw-2:19.0.0-1864.g66deef15.el9.x86_64 13/34 2024-04-04T20:50:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-04T20:50:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-04T20:50:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-04T20:50:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-04T20:50:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-04T20:50:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-04T20:50:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-04T20:50:21.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-04T20:50:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-04T20:50:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-04T20:50:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-04T20:50:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-04T20:50:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-04T20:50:21.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-04T20:50:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-04T20:50:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-04T20:50:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-04T20:50:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-2:19.0.0-1864.g66deef15.el9.x86_64 31/34 2024-04-04T20:50:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-04T20:50:21.264 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-2:19.0.0-1864.g66deef15.el9.x86_64 33/34 2024-04-04T20:50:22.335 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-04T20:50:22.335 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:22.335 INFO:teuthology.orchestra.run.smithi106.stdout:Upgraded: 2024-04-04T20:50:22.335 INFO:teuthology.orchestra.run.smithi106.stdout: librados2-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.335 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.335 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:50:22.335 INFO:teuthology.orchestra.run.smithi106.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-04T20:50:22.335 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.335 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.335 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.335 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.335 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-04T20:50:22.336 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-04T20:50:22.336 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-04T20:50:22.336 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.336 INFO:teuthology.orchestra.run.smithi106.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-04T20:50:22.336 INFO:teuthology.orchestra.run.smithi106.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-04T20:50:22.336 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-04T20:50:22.336 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.336 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-04T20:50:22.336 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.336 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-04T20:50:22.336 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-04T20:50:22.336 INFO:teuthology.orchestra.run.smithi106.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-04T20:50:22.336 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.337 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-common-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.337 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.337 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-04T20:50:22.337 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.337 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.337 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:22.337 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-04T20:50:22.337 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-04T20:50:22.337 INFO:teuthology.orchestra.run.smithi106.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-04T20:50:22.337 INFO:teuthology.orchestra.run.smithi106.stdout: re2-1:20211101-3.el9.x86_64 2024-04-04T20:50:22.337 INFO:teuthology.orchestra.run.smithi106.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-04T20:50:22.337 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:22.337 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:50:22.713 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-test 2024-04-04T20:50:23.274 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:28 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:50:23.428 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:50:23.429 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:50:23.429 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-04T20:50:23.429 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:50:23.429 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:50:23.429 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test x86_64 2:19.0.0-1864.g66deef15.el9 ceph 47 M 2024-04-04T20:50:23.429 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-04T20:50:23.429 INFO:teuthology.orchestra.run.smithi106.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-04T20:50:23.429 INFO:teuthology.orchestra.run.smithi106.stdout: libcephsqlite x86_64 2:19.0.0-1864.g66deef15.el9 ceph 163 k 2024-04-04T20:50:23.429 INFO:teuthology.orchestra.run.smithi106.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-04T20:50:23.429 INFO:teuthology.orchestra.run.smithi106.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-04T20:50:23.429 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-04T20:50:23.430 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:23.430 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:50:23.430 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:50:23.430 INFO:teuthology.orchestra.run.smithi106.stdout:Install 6 Packages 2024-04-04T20:50:23.430 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:23.430 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 48 M 2024-04-04T20:50:23.431 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 205 M 2024-04-04T20:50:23.431 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:50:23.726 INFO:teuthology.orchestra.run.smithi106.stdout:(1/6): libcephsqlite-19.0.0-1864.g66deef15.el9. 684 kB/s | 163 kB 00:00 2024-04-04T20:50:23.751 INFO:teuthology.orchestra.run.smithi106.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 824 kB/s | 217 kB 00:00 2024-04-04T20:50:23.810 INFO:teuthology.orchestra.run.smithi106.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-04-04T20:50:23.835 INFO:teuthology.orchestra.run.smithi106.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 748 kB/s | 63 kB 00:00 2024-04-04T20:50:23.969 INFO:teuthology.orchestra.run.smithi106.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-04-04T20:50:27.569 INFO:teuthology.orchestra.run.smithi106.stdout:(6/6): ceph-test-19.0.0-1864.g66deef15.el9.x86_ 12 MB/s | 47 MB 00:04 2024-04-04T20:50:27.572 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:50:27.572 INFO:teuthology.orchestra.run.smithi106.stdout:Total 12 MB/s | 48 MB 00:04 2024-04-04T20:50:27.587 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:50:27.608 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:50:27.608 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:50:27.994 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:50:27.995 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:50:28.581 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:50:28.652 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-04T20:50:28.693 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-04T20:50:28.734 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-04T20:50:28.778 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-04T20:50:28.805 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephsqlite-2:19.0.0-1864.g66deef15.el9.x86_64 5/6 2024-04-04T20:50:29.674 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1864.g66deef15.el9.x86_64 31/34 2024-04-04T20:50:29.674 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-04-04T20:50:29.674 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-04-04T20:50:29.674 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-04-04T20:50:29.674 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-04-04T20:50:29.675 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-04-04T20:50:29.675 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-04-04T20:50:29.675 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-04-04T20:50:29.675 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:30.611 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.0.0-1864.g66deef15.el9.x86_64 32/34 2024-04-04T20:50:30.636 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1864.g66deef15.el9.x86_64 32/34 2024-04-04T20:50:30.636 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:50:30.636 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-04T20:50:30.636 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-04T20:50:30.636 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-04T20:50:30.637 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:30.638 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-04T20:50:30.739 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-04T20:50:30.739 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-04T20:50:31.874 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-04T20:50:31.874 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.0.0-1864.g66deef15.el9.x86_64 1/34 2024-04-04T20:50:31.875 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.0.0-1864.g66deef15.el9.x86_64 2/34 2024-04-04T20:50:31.875 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.0.0-1864.g66deef15.el9.x86_64 3/34 2024-04-04T20:50:31.875 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.0.0-1864.g66deef15.el9.x86_64 4/34 2024-04-04T20:50:31.875 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.0.0-1864.g66deef15.el9.x86_64 5/34 2024-04-04T20:50:31.875 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.0.0-1864.g66deef15.el9.x86_6 6/34 2024-04-04T20:50:31.875 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.0.0-1864.g66deef15.el9.x86_64 7/34 2024-04-04T20:50:31.875 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1864.g66deef15.el9. 8/34 2024-04-04T20:50:31.875 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.0.0-1864.g66deef15.el9.x8 9/34 2024-04-04T20:50:31.875 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.0.0-1864.g66deef15.el9.x86_64 10/34 2024-04-04T20:50:31.875 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.0.0-1864.g66deef15.el9.x86_64 11/34 2024-04-04T20:50:31.875 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.0.0-1864.g66deef15.el9.x86_64 12/34 2024-04-04T20:50:31.875 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.0.0-1864.g66deef15.el9.x86_64 13/34 2024-04-04T20:50:31.875 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-04T20:50:31.876 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-04T20:50:31.876 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-04T20:50:31.876 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-04T20:50:31.876 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-04T20:50:31.876 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-04T20:50:31.876 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-04T20:50:31.876 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-04T20:50:31.876 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-04T20:50:31.876 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-04T20:50:31.876 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-04T20:50:31.876 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-04T20:50:31.876 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-04T20:50:31.876 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-04T20:50:31.877 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-04T20:50:31.877 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-04T20:50:31.877 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-04T20:50:31.877 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.0.0-1864.g66deef15.el9.x86_64 31/34 2024-04-04T20:50:31.878 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-04T20:50:31.878 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.0.0-1864.g66deef15.el9.x86_64 33/34 2024-04-04T20:50:32.914 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-04T20:50:32.914 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:32.914 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-04-04T20:50:32.914 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.914 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.914 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:50:32.915 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-04T20:50:32.915 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.915 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.915 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.915 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.915 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-04T20:50:32.915 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-04T20:50:32.915 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-04T20:50:32.915 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.915 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-04T20:50:32.915 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-04T20:50:32.915 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-04T20:50:32.915 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.915 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-04T20:50:32.916 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-3.el9.x86_64 2024-04-04T20:50:32.917 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-04T20:50:32.917 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:32.917 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:50:33.269 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1864.g66deef15.el9.x86_64 5/6 2024-04-04T20:50:33.275 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-test 2024-04-04T20:50:33.345 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-test-2:19.0.0-1864.g66deef15.el9.x86_64 6/6 2024-04-04T20:50:33.841 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:28 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:50:33.996 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:50:33.997 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:50:33.997 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-04T20:50:33.997 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:50:33.997 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:50:33.997 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.0.0-1864.g66deef15.el9 ceph 47 M 2024-04-04T20:50:33.997 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-04T20:50:33.997 INFO:teuthology.orchestra.run.smithi203.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-04T20:50:33.997 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.0.0-1864.g66deef15.el9 ceph 163 k 2024-04-04T20:50:33.997 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-04T20:50:33.997 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-04T20:50:33.997 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-04T20:50:33.997 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:33.998 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:50:33.998 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:50:33.998 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-04-04T20:50:33.998 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:33.998 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 48 M 2024-04-04T20:50:33.999 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 205 M 2024-04-04T20:50:33.999 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:50:34.309 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-04-04T20:50:34.359 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-04-04T20:50:34.392 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-04-04T20:50:34.418 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): libcephsqlite-19.0.0-1864.g66deef15.el9. 575 kB/s | 163 kB 00:00 2024-04-04T20:50:34.751 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 522 kB/s | 187 kB 00:00 2024-04-04T20:50:35.824 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-test-2:19.0.0-1864.g66deef15.el9.x86_64 6/6 2024-04-04T20:50:35.824 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-test-2:19.0.0-1864.g66deef15.el9.x86_64 1/6 2024-04-04T20:50:35.824 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephsqlite-2:19.0.0-1864.g66deef15.el9.x86_64 2/6 2024-04-04T20:50:35.824 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-04T20:50:35.824 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-04T20:50:35.824 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-04T20:50:36.198 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-04T20:50:36.198 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:36.198 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:50:36.198 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:36.198 INFO:teuthology.orchestra.run.smithi106.stdout: jq-1.6-16.el9.x86_64 2024-04-04T20:50:36.198 INFO:teuthology.orchestra.run.smithi106.stdout: libcephsqlite-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:36.198 INFO:teuthology.orchestra.run.smithi106.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-04T20:50:36.198 INFO:teuthology.orchestra.run.smithi106.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-04T20:50:36.198 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-04T20:50:36.198 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:36.199 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:50:36.460 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): ceph-test-19.0.0-1864.g66deef15.el9.x86_ 20 MB/s | 47 MB 00:02 2024-04-04T20:50:36.463 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:50:36.463 INFO:teuthology.orchestra.run.smithi203.stdout:Total 20 MB/s | 48 MB 00:02 2024-04-04T20:50:36.478 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:50:36.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph 2024-04-04T20:50:36.500 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:50:36.500 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:50:36.893 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:50:36.894 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:50:37.058 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:42 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:50:37.222 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:50:37.224 INFO:teuthology.orchestra.run.smithi106.stdout:========================================================================================= 2024-04-04T20:50:37.224 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-04T20:50:37.224 INFO:teuthology.orchestra.run.smithi106.stdout:========================================================================================= 2024-04-04T20:50:37.224 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:50:37.225 INFO:teuthology.orchestra.run.smithi106.stdout: ceph x86_64 2:19.0.0-1864.g66deef15.el9 ceph 6.6 k 2024-04-04T20:50:37.225 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-04T20:50:37.225 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds x86_64 2:19.0.0-1864.g66deef15.el9 ceph 2.1 M 2024-04-04T20:50:37.225 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr x86_64 2:19.0.0-1864.g66deef15.el9 ceph 1.5 M 2024-04-04T20:50:37.225 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 248 k 2024-04-04T20:50:37.225 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon x86_64 2:19.0.0-1864.g66deef15.el9 ceph 4.7 M 2024-04-04T20:50:37.225 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd x86_64 2:19.0.0-1864.g66deef15.el9 ceph 17 M 2024-04-04T20:50:37.225 INFO:teuthology.orchestra.run.smithi106.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-04T20:50:37.225 INFO:teuthology.orchestra.run.smithi106.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-04T20:50:37.225 INFO:teuthology.orchestra.run.smithi106.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-04T20:50:37.225 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-04T20:50:37.225 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-04T20:50:37.225 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-04T20:50:37.226 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-04T20:50:37.226 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-04T20:50:37.226 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-04T20:50:37.226 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-04T20:50:37.226 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-04T20:50:37.226 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-04T20:50:37.226 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-04T20:50:37.226 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-04T20:50:37.226 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-04T20:50:37.226 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-04T20:50:37.226 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-04T20:50:37.226 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-04T20:50:37.226 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-04T20:50:37.226 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout:========================================================================================= 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout:Install 36 Packages 2024-04-04T20:50:37.227 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:37.228 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 30 M 2024-04-04T20:50:37.228 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 107 M 2024-04-04T20:50:37.228 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:50:37.465 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:50:37.537 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-04T20:50:37.575 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-04T20:50:37.617 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-04T20:50:37.621 INFO:teuthology.orchestra.run.smithi106.stdout:(1/36): ceph-19.0.0-1864.g66deef15.el9.x86_64.r 32 kB/s | 6.6 kB 00:00 2024-04-04T20:50:37.660 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-04T20:50:37.688 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.0.0-1864.g66deef15.el9.x86_64 5/6 2024-04-04T20:50:37.730 INFO:teuthology.orchestra.run.smithi106.stdout:(2/36): ceph-mgr-19.0.0-1864.g66deef15.el9.x86_ 4.8 MB/s | 1.5 MB 00:00 2024-04-04T20:50:38.130 INFO:teuthology.orchestra.run.smithi106.stdout:(3/36): ceph-mds-19.0.0-1864.g66deef15.el9.x86_ 3.0 MB/s | 2.1 MB 00:00 2024-04-04T20:50:38.244 INFO:teuthology.orchestra.run.smithi106.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1864.g66de 2.2 MB/s | 248 kB 00:00 2024-04-04T20:50:38.348 INFO:teuthology.orchestra.run.smithi106.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-04-04T20:50:38.389 INFO:teuthology.orchestra.run.smithi106.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-04T20:50:38.423 INFO:teuthology.orchestra.run.smithi106.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2024-04-04T20:50:38.456 INFO:teuthology.orchestra.run.smithi106.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-04T20:50:38.615 INFO:teuthology.orchestra.run.smithi106.stdout:(9/36): ceph-osd-19.0.0-1864.g66deef15.el9.x86_ 19 MB/s | 17 MB 00:00 2024-04-04T20:50:38.632 INFO:teuthology.orchestra.run.smithi106.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 220 kB/s | 39 kB 00:00 2024-04-04T20:50:38.657 INFO:teuthology.orchestra.run.smithi106.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-04-04T20:50:38.699 INFO:teuthology.orchestra.run.smithi106.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-04-04T20:50:38.724 INFO:teuthology.orchestra.run.smithi106.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-04-04T20:50:38.750 INFO:teuthology.orchestra.run.smithi106.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-04-04T20:50:38.767 INFO:teuthology.orchestra.run.smithi106.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-04-04T20:50:38.884 INFO:teuthology.orchestra.run.smithi106.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-04-04T20:50:38.934 INFO:teuthology.orchestra.run.smithi106.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 680 kB/s | 125 kB 00:00 2024-04-04T20:50:39.018 INFO:teuthology.orchestra.run.smithi106.stdout:(18/36): ceph-mon-19.0.0-1864.g66deef15.el9.x86 3.4 MB/s | 4.7 MB 00:01 2024-04-04T20:50:39.285 INFO:teuthology.orchestra.run.smithi106.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-04-04T20:50:39.376 INFO:teuthology.orchestra.run.smithi106.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 307 kB/s | 151 kB 00:00 2024-04-04T20:50:39.435 INFO:teuthology.orchestra.run.smithi106.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 412 kB/s | 172 kB 00:00 2024-04-04T20:50:39.560 INFO:teuthology.orchestra.run.smithi106.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 58 kB/s | 11 kB 00:00 2024-04-04T20:50:39.702 INFO:teuthology.orchestra.run.smithi106.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 66 kB/s | 18 kB 00:00 2024-04-04T20:50:39.811 INFO:teuthology.orchestra.run.smithi106.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 681 kB/s | 358 kB 00:00 2024-04-04T20:50:39.836 INFO:teuthology.orchestra.run.smithi106.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 84 kB/s | 23 kB 00:00 2024-04-04T20:50:39.994 INFO:teuthology.orchestra.run.smithi106.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 67 kB/s | 19 kB 00:00 2024-04-04T20:50:40.078 INFO:teuthology.orchestra.run.smithi106.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 74 kB/s | 20 kB 00:00 2024-04-04T20:50:40.178 INFO:teuthology.orchestra.run.smithi106.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 135 kB/s | 46 kB 00:00 2024-04-04T20:50:40.345 INFO:teuthology.orchestra.run.smithi106.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 225 kB/s | 79 kB 00:00 2024-04-04T20:50:40.445 INFO:teuthology.orchestra.run.smithi106.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 61 kB/s | 16 kB 00:00 2024-04-04T20:50:40.604 INFO:teuthology.orchestra.run.smithi106.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 518 kB/s | 272 kB 00:00 2024-04-04T20:50:40.721 INFO:teuthology.orchestra.run.smithi106.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 130 kB/s | 36 kB 00:00 2024-04-04T20:50:40.788 INFO:teuthology.orchestra.run.smithi106.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 203 kB/s | 90 kB 00:00 2024-04-04T20:50:41.055 INFO:teuthology.orchestra.run.smithi106.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 75 kB/s | 20 kB 00:00 2024-04-04T20:50:41.147 INFO:teuthology.orchestra.run.smithi106.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 423 kB/s | 230 kB 00:00 2024-04-04T20:50:41.305 INFO:teuthology.orchestra.run.smithi106.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 732 kB/s | 427 kB 00:00 2024-04-04T20:50:41.309 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:50:41.310 INFO:teuthology.orchestra.run.smithi106.stdout:Total 7.3 MB/s | 30 MB 00:04 2024-04-04T20:50:41.335 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:50:41.381 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:50:41.381 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:50:41.813 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:50:41.814 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:50:42.191 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1864.g66deef15.el9.x86_64 5/6 2024-04-04T20:50:42.263 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.0.0-1864.g66deef15.el9.x86_64 6/6 2024-04-04T20:50:43.143 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:50:43.216 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-04T20:50:43.249 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-04T20:50:43.277 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-04T20:50:43.375 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-04T20:50:43.451 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-04T20:50:43.483 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-04T20:50:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-04T20:50:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-04T20:50:43.694 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-04T20:50:43.765 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-04T20:50:43.820 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-04T20:50:43.859 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-04T20:50:43.907 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-04T20:50:43.963 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-04T20:50:44.156 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-04T20:50:44.204 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-04T20:50:44.235 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-04T20:50:44.269 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-04T20:50:44.299 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-04T20:50:44.375 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-04T20:50:44.409 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-04T20:50:44.443 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-04T20:50:44.483 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-04T20:50:44.518 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-04T20:50:44.553 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-04T20:50:44.642 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-04T20:50:44.676 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-04T20:50:44.739 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-04T20:50:44.744 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-test-2:19.0.0-1864.g66deef15.el9.x86_64 6/6 2024-04-04T20:50:44.744 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.0.0-1864.g66deef15.el9.x86_64 1/6 2024-04-04T20:50:44.744 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.0.0-1864.g66deef15.el9.x86_64 2/6 2024-04-04T20:50:44.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-04T20:50:44.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-04T20:50:44.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-04T20:50:44.875 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-04T20:50:44.993 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-04T20:50:45.085 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-04T20:50:45.085 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:45.085 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:50:45.085 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:45.085 INFO:teuthology.orchestra.run.smithi203.stdout: jq-1.6-16.el9.x86_64 2024-04-04T20:50:45.085 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:45.085 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-04T20:50:45.085 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-04T20:50:45.085 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-04T20:50:45.085 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:45.085 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:50:45.153 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1864.g66deef15.el9. 31/36 2024-04-04T20:50:45.180 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-2:19.0.0-1864.g66deef15.el9.x86_64 32/36 2024-04-04T20:50:45.206 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1864.g66deef15.el9.x86_64 32/36 2024-04-04T20:50:45.206 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:50:45.206 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-04T20:50:45.206 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-04T20:50:45.206 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-04T20:50:45.207 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:45.382 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph 2024-04-04T20:50:45.947 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:40 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:50:46.100 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:50:46.102 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-04-04T20:50:46.102 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-04T20:50:46.102 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-04-04T20:50:46.102 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:50:46.102 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.0.0-1864.g66deef15.el9 ceph 6.6 k 2024-04-04T20:50:46.102 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-04T20:50:46.103 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.0.0-1864.g66deef15.el9 ceph 2.1 M 2024-04-04T20:50:46.103 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.0.0-1864.g66deef15.el9 ceph 1.5 M 2024-04-04T20:50:46.103 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 248 k 2024-04-04T20:50:46.103 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.0.0-1864.g66deef15.el9 ceph 4.7 M 2024-04-04T20:50:46.103 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.0.0-1864.g66deef15.el9 ceph 17 M 2024-04-04T20:50:46.103 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-04T20:50:46.103 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-04T20:50:46.103 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-04T20:50:46.103 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-04T20:50:46.103 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-04T20:50:46.103 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-04T20:50:46.103 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-04T20:50:46.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-04T20:50:46.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-04T20:50:46.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-04T20:50:46.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-04T20:50:46.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-04T20:50:46.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-04T20:50:46.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-04T20:50:46.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-04T20:50:46.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-04T20:50:46.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-04T20:50:46.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-04T20:50:46.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-04T20:50:46.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-04T20:50:46.105 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-04T20:50:46.105 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-04T20:50:46.105 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-04T20:50:46.105 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-04T20:50:46.105 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-04T20:50:46.105 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-04T20:50:46.105 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-04T20:50:46.105 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-04T20:50:46.105 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-04T20:50:46.105 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-04T20:50:46.105 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:46.105 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:50:46.105 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-04-04T20:50:46.105 INFO:teuthology.orchestra.run.smithi203.stdout:Install 36 Packages 2024-04-04T20:50:46.106 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:46.106 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 30 M 2024-04-04T20:50:46.106 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 107 M 2024-04-04T20:50:46.107 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:50:46.648 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-osd-2:19.0.0-1864.g66deef15.el9.x86_64 33/36 2024-04-04T20:50:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-osd-2:19.0.0-1864.g66deef15.el9.x86_64 33/36 2024-04-04T20:50:46.674 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:50:46.674 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-04T20:50:46.674 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-04T20:50:46.674 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-04T20:50:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:46.715 INFO:teuthology.orchestra.run.smithi203.stdout:(1/36): ceph-19.0.0-1864.g66deef15.el9.x86_64.r 29 kB/s | 6.6 kB 00:00 2024-04-04T20:50:46.857 INFO:teuthology.orchestra.run.smithi203.stdout:(2/36): ceph-mds-19.0.0-1864.g66deef15.el9.x86_ 5.8 MB/s | 2.1 MB 00:00 2024-04-04T20:50:47.115 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mon-2:19.0.0-1864.g66deef15.el9.x86_64 34/36 2024-04-04T20:50:47.139 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mon-2:19.0.0-1864.g66deef15.el9.x86_64 34/36 2024-04-04T20:50:47.139 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:50:47.139 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-04T20:50:47.139 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-04T20:50:47.139 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-04T20:50:47.139 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:47.166 INFO:teuthology.orchestra.run.smithi203.stdout:(3/36): ceph-mgr-19.0.0-1864.g66deef15.el9.x86_ 2.2 MB/s | 1.5 MB 00:00 2024-04-04T20:50:47.341 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mds-2:19.0.0-1864.g66deef15.el9.x86_64 35/36 2024-04-04T20:50:47.364 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mds-2:19.0.0-1864.g66deef15.el9.x86_64 35/36 2024-04-04T20:50:47.364 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:50:47.365 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-04T20:50:47.365 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-04T20:50:47.365 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-04T20:50:47.365 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:47.421 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-2:19.0.0-1864.g66deef15.el9.x86_64 36/36 2024-04-04T20:50:47.716 INFO:teuthology.orchestra.run.smithi203.stdout:(4/36): ceph-osd-19.0.0-1864.g66deef15.el9.x86_ 20 MB/s | 17 MB 00:00 2024-04-04T20:50:47.758 INFO:teuthology.orchestra.run.smithi203.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1864.g66de 419 kB/s | 248 kB 00:00 2024-04-04T20:50:47.883 INFO:teuthology.orchestra.run.smithi203.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-04-04T20:50:47.900 INFO:teuthology.orchestra.run.smithi203.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 144 kB 00:00 2024-04-04T20:50:47.950 INFO:teuthology.orchestra.run.smithi203.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 471 kB/s | 32 kB 00:00 2024-04-04T20:50:47.976 INFO:teuthology.orchestra.run.smithi203.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 627 kB/s | 47 kB 00:00 2024-04-04T20:50:48.023 INFO:teuthology.orchestra.run.smithi203.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 533 kB/s | 39 kB 00:00 2024-04-04T20:50:48.043 INFO:teuthology.orchestra.run.smithi203.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-04-04T20:50:48.093 INFO:teuthology.orchestra.run.smithi203.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 16 MB/s | 1.1 MB 00:00 2024-04-04T20:50:48.119 INFO:teuthology.orchestra.run.smithi203.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-04-04T20:50:48.144 INFO:teuthology.orchestra.run.smithi203.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-04-04T20:50:48.161 INFO:teuthology.orchestra.run.smithi203.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-04-04T20:50:48.303 INFO:teuthology.orchestra.run.smithi203.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-04-04T20:50:48.370 INFO:teuthology.orchestra.run.smithi203.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 555 kB/s | 125 kB 00:00 2024-04-04T20:50:48.437 INFO:teuthology.orchestra.run.smithi203.stdout:(18/36): ceph-mon-19.0.0-1864.g66deef15.el9.x86 2.7 MB/s | 4.7 MB 00:01 2024-04-04T20:50:48.671 INFO:teuthology.orchestra.run.smithi203.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 144 kB/s | 43 kB 00:00 2024-04-04T20:50:48.754 INFO:teuthology.orchestra.run.smithi203.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 335 kB/s | 151 kB 00:00 2024-04-04T20:50:48.846 INFO:teuthology.orchestra.run.smithi203.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-04-04T20:50:48.888 INFO:teuthology.orchestra.run.smithi203.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 381 kB/s | 172 kB 00:00 2024-04-04T20:50:48.947 INFO:teuthology.orchestra.run.smithi203.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-04-04T20:50:48.981 INFO:teuthology.orchestra.run.smithi203.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-04-04T20:50:49.014 INFO:teuthology.orchestra.run.smithi203.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-04-04T20:50:49.039 INFO:teuthology.orchestra.run.smithi203.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-04-04T20:50:49.081 INFO:teuthology.orchestra.run.smithi203.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-04-04T20:50:49.124 INFO:teuthology.orchestra.run.smithi203.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 423 kB/s | 46 kB 00:00 2024-04-04T20:50:49.157 INFO:teuthology.orchestra.run.smithi203.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 671 kB/s | 79 kB 00:00 2024-04-04T20:50:49.216 INFO:teuthology.orchestra.run.smithi203.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2024-04-04T20:50:49.258 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-2:19.0.0-1864.g66deef15.el9.x86_64 36/36 2024-04-04T20:50:49.258 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-2:19.0.0-1864.g66deef15.el9.x86_64 1/36 2024-04-04T20:50:49.258 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mds-2:19.0.0-1864.g66deef15.el9.x86_64 2/36 2024-04-04T20:50:49.258 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-2:19.0.0-1864.g66deef15.el9.x86_64 3/36 2024-04-04T20:50:49.259 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mon-2:19.0.0-1864.g66deef15.el9.x86_64 4/36 2024-04-04T20:50:49.259 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-osd-2:19.0.0-1864.g66deef15.el9.x86_64 5/36 2024-04-04T20:50:49.259 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1864.g66deef15.el9. 6/36 2024-04-04T20:50:49.260 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-04T20:50:49.260 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-04T20:50:49.260 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-04T20:50:49.260 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-04T20:50:49.260 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-04T20:50:49.260 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-04T20:50:49.261 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-04T20:50:49.261 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-04T20:50:49.261 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-04T20:50:49.261 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-04T20:50:49.261 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-04T20:50:49.261 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-04T20:50:49.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-04T20:50:49.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-04T20:50:49.262 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-04T20:50:49.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-04T20:50:49.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-04T20:50:49.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-04T20:50:49.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-04T20:50:49.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-04T20:50:49.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-04T20:50:49.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-04T20:50:49.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-04T20:50:49.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-04T20:50:49.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-04T20:50:49.263 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-04T20:50:49.264 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-04T20:50:49.264 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-04T20:50:49.264 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-04T20:50:49.264 INFO:teuthology.orchestra.run.smithi203.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-04-04T20:50:49.283 INFO:teuthology.orchestra.run.smithi203.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 716 kB/s | 90 kB 00:00 2024-04-04T20:50:49.317 INFO:teuthology.orchestra.run.smithi203.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 355 kB/s | 36 kB 00:00 2024-04-04T20:50:49.375 INFO:teuthology.orchestra.run.smithi203.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-04-04T20:50:49.409 INFO:teuthology.orchestra.run.smithi203.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2024-04-04T20:50:49.542 INFO:teuthology.orchestra.run.smithi203.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2024-04-04T20:50:49.547 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:50:49.547 INFO:teuthology.orchestra.run.smithi203.stdout:Total 8.6 MB/s | 30 MB 00:03 2024-04-04T20:50:49.573 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:50:49.616 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:50:49.616 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:50:50.021 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:50:50.021 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:50:50.233 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-04T20:50:50.233 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:50.233 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:50:50.234 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:50.234 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:50.234 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:50.234 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:50:50.234 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:50.234 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:50.234 INFO:teuthology.orchestra.run.smithi106.stdout: lua-5.4.4-4.el9.x86_64 2024-04-04T20:50:50.234 INFO:teuthology.orchestra.run.smithi106.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-04T20:50:50.234 INFO:teuthology.orchestra.run.smithi106.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-04T20:50:50.234 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-04T20:50:50.234 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-04T20:50:50.234 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-04T20:50:50.234 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-04T20:50:50.235 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-04T20:50:50.235 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-04T20:50:50.235 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-04T20:50:50.235 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-04T20:50:50.235 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-04T20:50:50.235 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-04T20:50:50.235 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-04T20:50:50.235 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-04T20:50:50.235 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-04T20:50:50.235 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-04T20:50:50.235 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-04T20:50:50.235 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ply-3.11-14.el9.noarch 2024-04-04T20:50:50.235 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-04T20:50:50.235 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-04T20:50:50.236 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-04T20:50:50.236 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-04T20:50:50.236 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-04T20:50:50.236 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-04T20:50:50.236 INFO:teuthology.orchestra.run.smithi106.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-04T20:50:50.236 INFO:teuthology.orchestra.run.smithi106.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-04T20:50:50.236 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-04T20:50:50.236 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-04T20:50:50.236 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-04T20:50:50.236 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:50.236 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:50:50.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-base 2024-04-04T20:50:51.138 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:56 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:50:51.235 INFO:teuthology.orchestra.run.smithi106.stdout:Package ceph-base-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:50:51.287 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:50:51.288 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-04T20:50:51.288 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:50:51.348 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install cephadm 2024-04-04T20:50:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:50:51.453 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-04T20:50:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-04T20:50:51.514 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-04T20:50:51.584 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-04T20:50:51.638 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-04T20:50:51.671 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-04T20:50:51.762 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-04T20:50:51.832 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-04T20:50:51.874 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-04T20:50:51.910 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:56 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:50:51.945 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-04T20:50:51.999 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-04T20:50:52.038 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-04T20:50:52.058 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:50:52.059 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:50:52.059 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-04T20:50:52.059 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:50:52.059 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:50:52.060 INFO:teuthology.orchestra.run.smithi106.stdout: cephadm noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 749 k 2024-04-04T20:50:52.060 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:52.060 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:50:52.060 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:50:52.060 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:50:52.060 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:52.060 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 749 k 2024-04-04T20:50:52.060 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 755 k 2024-04-04T20:50:52.060 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:50:52.087 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-04T20:50:52.142 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-04T20:50:52.331 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-04T20:50:52.357 INFO:teuthology.orchestra.run.smithi106.stdout:cephadm-19.0.0-1864.g66deef15.el9.noarch.rpm 2.5 MB/s | 749 kB 00:00 2024-04-04T20:50:52.357 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:50:52.357 INFO:teuthology.orchestra.run.smithi106.stdout:Total 2.5 MB/s | 749 kB 00:00 2024-04-04T20:50:52.357 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:50:52.363 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:50:52.363 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:50:52.369 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:50:52.369 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:50:52.384 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-04T20:50:52.414 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-04T20:50:52.445 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:50:52.449 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-04T20:50:52.480 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-04T20:50:52.555 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-04T20:50:52.590 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-04T20:50:52.640 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-04T20:50:52.680 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-04T20:50:52.715 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-04T20:50:52.742 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-04T20:50:52.832 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-04T20:50:52.878 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-04T20:50:52.901 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cephadm-2:19.0.0-1864.g66deef15.el9.noarch 1/1 2024-04-04T20:50:52.954 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-04T20:50:53.010 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephadm-2:19.0.0-1864.g66deef15.el9.noarch 1/1 2024-04-04T20:50:53.091 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-04T20:50:53.215 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-04T20:50:53.382 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1864.g66deef15.el9. 31/36 2024-04-04T20:50:53.412 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.0.0-1864.g66deef15.el9.x86_64 32/36 2024-04-04T20:50:53.436 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1864.g66deef15.el9.x86_64 32/36 2024-04-04T20:50:53.437 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:50:53.437 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-04T20:50:53.437 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-04T20:50:53.437 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-04T20:50:53.437 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:53.450 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cephadm-2:19.0.0-1864.g66deef15.el9.noarch 1/1 2024-04-04T20:50:53.758 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephadm-2:19.0.0-1864.g66deef15.el9.noarch 1/1 2024-04-04T20:50:53.758 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:53.758 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:50:53.758 INFO:teuthology.orchestra.run.smithi106.stdout: cephadm-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:50:53.758 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:53.758 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:50:53.951 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-immutable-object-cache 2024-04-04T20:50:54.521 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:59 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:50:54.673 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:50:54.674 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:50:54.674 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:50:54.674 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:50:54.674 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:50:54.674 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1864.g66deef15.el9 ceph 145 k 2024-04-04T20:50:54.675 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:54.675 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:50:54.675 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:50:54.675 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:50:54.675 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:54.675 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 145 k 2024-04-04T20:50:54.675 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 436 k 2024-04-04T20:50:54.675 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:50:54.879 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.0.0-1864.g66deef15.el9.x86_64 33/36 2024-04-04T20:50:54.894 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-immutable-object-cache-19.0.0-1864.g66deef 662 kB/s | 145 kB 00:00 2024-04-04T20:50:54.895 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:50:54.895 INFO:teuthology.orchestra.run.smithi106.stdout:Total 657 kB/s | 145 kB 00:00 2024-04-04T20:50:54.895 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:50:54.903 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:50:54.903 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:50:54.906 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.0.0-1864.g66deef15.el9.x86_64 33/36 2024-04-04T20:50:54.906 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:50:54.906 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-04T20:50:54.906 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-04T20:50:54.906 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-04T20:50:54.906 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:54.956 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:50:54.956 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:50:55.091 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:50:55.162 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1864.g66deef15. 1/1 2024-04-04T20:50:55.186 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1864.g66deef15. 1/1 2024-04-04T20:50:55.186 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:50:55.186 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-04T20:50:55.186 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:55.347 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.0.0-1864.g66deef15.el9.x86_64 34/36 2024-04-04T20:50:55.370 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.0.0-1864.g66deef15.el9.x86_64 34/36 2024-04-04T20:50:55.370 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:50:55.370 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-04T20:50:55.370 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-04T20:50:55.370 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-04T20:50:55.370 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:55.572 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.0.0-1864.g66deef15.el9.x86_64 35/36 2024-04-04T20:50:55.595 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.0.0-1864.g66deef15.el9.x86_64 35/36 2024-04-04T20:50:55.595 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:50:55.596 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-04T20:50:55.596 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-04T20:50:55.596 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-04T20:50:55.596 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:55.661 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.0.0-1864.g66deef15.el9.x86_64 36/36 2024-04-04T20:50:56.150 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1864.g66deef15. 1/1 2024-04-04T20:50:56.150 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:56.150 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:50:56.151 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:56.151 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:56.151 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:50:56.392 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr 2024-04-04T20:50:56.964 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:01 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:50:57.063 INFO:teuthology.orchestra.run.smithi106.stdout:Package ceph-mgr-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:50:57.118 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:50:57.118 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-04T20:50:57.118 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:50:57.181 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr-dashboard 2024-04-04T20:50:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-2:19.0.0-1864.g66deef15.el9.x86_64 36/36 2024-04-04T20:50:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.0.0-1864.g66deef15.el9.x86_64 1/36 2024-04-04T20:50:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.0.0-1864.g66deef15.el9.x86_64 2/36 2024-04-04T20:50:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.0.0-1864.g66deef15.el9.x86_64 3/36 2024-04-04T20:50:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.0.0-1864.g66deef15.el9.x86_64 4/36 2024-04-04T20:50:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.0.0-1864.g66deef15.el9.x86_64 5/36 2024-04-04T20:50:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1864.g66deef15.el9. 6/36 2024-04-04T20:50:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-04T20:50:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-04T20:50:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-04T20:50:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-04T20:50:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-04T20:50:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-04T20:50:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-04T20:50:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-04T20:50:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-04T20:50:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-04T20:50:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-04T20:50:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-04T20:50:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-04T20:50:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-04T20:50:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-04T20:50:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-04T20:50:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-04T20:50:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-04T20:50:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-04T20:50:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-04T20:50:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-04T20:50:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-04T20:50:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-04T20:50:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-04T20:50:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-04T20:50:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-04T20:50:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-04T20:50:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-04T20:50:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-04T20:50:57.741 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:02 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:50:57.890 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:50:57.891 INFO:teuthology.orchestra.run.smithi106.stdout:====================================================================================== 2024-04-04T20:50:57.891 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-04T20:50:57.891 INFO:teuthology.orchestra.run.smithi106.stdout:====================================================================================== 2024-04-04T20:50:57.891 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:50:57.891 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 1.8 M 2024-04-04T20:50:57.891 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-04T20:50:57.891 INFO:teuthology.orchestra.run.smithi106.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-04T20:50:57.891 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 31 k 2024-04-04T20:50:57.892 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 16 k 2024-04-04T20:50:57.892 INFO:teuthology.orchestra.run.smithi106.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-04T20:50:57.892 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-04T20:50:57.892 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-04T20:50:57.892 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-04T20:50:57.892 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-04T20:50:57.892 INFO:teuthology.orchestra.run.smithi106.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-04T20:50:57.892 INFO:teuthology.orchestra.run.smithi106.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-04T20:50:57.892 INFO:teuthology.orchestra.run.smithi106.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-04T20:50:57.892 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:57.892 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:50:57.892 INFO:teuthology.orchestra.run.smithi106.stdout:====================================================================================== 2024-04-04T20:50:57.892 INFO:teuthology.orchestra.run.smithi106.stdout:Install 12 Packages 2024-04-04T20:50:57.892 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:50:57.893 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 6.9 M 2024-04-04T20:50:57.893 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 26 M 2024-04-04T20:50:57.893 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:50:58.248 INFO:teuthology.orchestra.run.smithi106.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1864.g66 167 kB/s | 31 kB 00:00 2024-04-04T20:50:58.348 INFO:teuthology.orchestra.run.smithi106.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1864.g66d 57 kB/s | 16 kB 00:00 2024-04-04T20:50:58.398 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-04T20:50:58.399 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:58.399 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:50:58.399 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:58.399 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:58.399 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:58.399 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:50:58.399 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:58.399 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:50:58.399 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-04-04T20:50:58.399 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-04T20:50:58.399 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-04T20:50:58.400 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-04T20:50:58.400 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-04T20:50:58.400 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-04T20:50:58.400 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-04T20:50:58.400 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-04T20:50:58.400 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-04T20:50:58.400 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-04T20:50:58.400 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-04T20:50:58.400 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-04T20:50:58.400 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-04T20:50:58.400 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-04T20:50:58.400 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-04T20:50:58.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-04T20:50:58.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-04T20:50:58.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-04T20:50:58.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-04-04T20:50:58.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-04T20:50:58.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-04T20:50:58.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-04T20:50:58.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-04T20:50:58.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-04T20:50:58.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-04T20:50:58.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-04T20:50:58.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-04T20:50:58.402 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-04T20:50:58.402 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-04T20:50:58.402 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-04T20:50:58.402 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:50:58.402 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:50:58.415 INFO:teuthology.orchestra.run.smithi106.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-04-04T20:50:58.432 INFO:teuthology.orchestra.run.smithi106.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-04-04T20:50:58.566 INFO:teuthology.orchestra.run.smithi106.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1864.g66deef1 3.5 MB/s | 1.8 MB 00:00 2024-04-04T20:50:58.633 INFO:teuthology.orchestra.run.smithi106.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.9 MB/s | 863 kB 00:00 2024-04-04T20:50:58.683 INFO:teuthology.orchestra.run.smithi106.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-04-04T20:50:58.750 INFO:teuthology.orchestra.run.smithi106.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.7 MB/s | 548 kB 00:00 2024-04-04T20:50:58.753 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base 2024-04-04T20:50:58.792 INFO:teuthology.orchestra.run.smithi106.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 741 kB/s | 31 kB 00:00 2024-04-04T20:50:58.834 INFO:teuthology.orchestra.run.smithi106.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 960 kB/s | 144 kB 00:00 2024-04-04T20:50:58.884 INFO:teuthology.orchestra.run.smithi106.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-04-04T20:50:59.059 INFO:teuthology.orchestra.run.smithi106.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.8 MB/s | 2.0 MB 00:00 2024-04-04T20:50:59.062 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:50:59.062 INFO:teuthology.orchestra.run.smithi106.stdout:Total 5.9 MB/s | 6.9 MB 00:01 2024-04-04T20:50:59.080 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:50:59.097 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:50:59.097 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:50:59.258 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:50:59.259 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:50:59.324 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:54 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:50:59.422 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-base-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:50:59.474 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:50:59.475 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-04T20:50:59.475 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:50:59.543 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install cephadm 2024-04-04T20:50:59.775 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:50:59.851 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-04T20:50:59.886 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-04T20:50:59.933 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-04T20:50:59.972 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-04T20:51:00.080 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-04T20:51:00.101 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:55 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:00.204 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-04T20:51:00.251 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:00.252 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:00.252 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-04T20:51:00.253 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:00.253 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:51:00.253 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 749 k 2024-04-04T20:51:00.253 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:00.253 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:51:00.253 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:00.253 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:51:00.253 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:00.253 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 749 k 2024-04-04T20:51:00.254 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 755 k 2024-04-04T20:51:00.254 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:51:00.299 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-04T20:51:00.342 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-04T20:51:00.371 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-04T20:51:00.403 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1864.g66deef15.el9 10/12 2024-04-04T20:51:00.523 INFO:teuthology.orchestra.run.smithi203.stdout:cephadm-19.0.0-1864.g66deef15.el9.noarch.rpm 2.7 MB/s | 749 kB 00:00 2024-04-04T20:51:00.524 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:00.524 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.7 MB/s | 749 kB 00:00 2024-04-04T20:51:00.524 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:51:00.529 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:51:00.529 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:51:00.535 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:51:00.535 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:51:00.618 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:51:00.647 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1864.g66deef15.el 11/12 2024-04-04T20:51:00.678 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1864.g66deef15.el9.noa 12/12 2024-04-04T20:51:01.093 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-1864.g66deef15.el9.noarch 1/1 2024-04-04T20:51:01.186 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.0.0-1864.g66deef15.el9.noarch 1/1 2024-04-04T20:51:01.506 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1864.g66deef15.el9.noa 12/12 2024-04-04T20:51:01.507 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1864.g66deef15.el 1/12 2024-04-04T20:51:01.507 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1864.g66deef15.el9.noa 2/12 2024-04-04T20:51:01.507 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1864.g66deef15.el9 3/12 2024-04-04T20:51:01.507 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-04T20:51:01.507 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-04T20:51:01.507 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-04T20:51:01.507 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-04T20:51:01.507 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-04T20:51:01.507 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-04T20:51:01.507 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-04T20:51:01.507 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-04T20:51:01.562 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-1864.g66deef15.el9.noarch 1/1 2024-04-04T20:51:01.841 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-04T20:51:01.841 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:01.841 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:51:01.841 INFO:teuthology.orchestra.run.smithi106.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-04T20:51:01.841 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-grafana-dashboards-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:01.841 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:01.841 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-prometheus-alerts-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:01.841 INFO:teuthology.orchestra.run.smithi106.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-04T20:51:01.841 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-04T20:51:01.842 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-04T20:51:01.842 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-04T20:51:01.842 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-04T20:51:01.842 INFO:teuthology.orchestra.run.smithi106.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-04T20:51:01.842 INFO:teuthology.orchestra.run.smithi106.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-04T20:51:01.842 INFO:teuthology.orchestra.run.smithi106.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-04T20:51:01.842 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:01.842 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:01.847 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.0.0-1864.g66deef15.el9.noarch 1/1 2024-04-04T20:51:01.847 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:01.847 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:51:01.848 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:01.848 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:01.848 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:02.049 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-immutable-object-cache 2024-04-04T20:51:02.081 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-04T20:51:02.628 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:57 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:02.644 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:07 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:02.781 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:02.782 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:02.782 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:51:02.782 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:02.782 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:51:02.782 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1864.g66deef15.el9 ceph 145 k 2024-04-04T20:51:02.782 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:02.782 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:51:02.782 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:02.782 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:51:02.782 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:02.783 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 145 k 2024-04-04T20:51:02.783 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 436 k 2024-04-04T20:51:02.783 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:51:02.795 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:02.796 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-04-04T20:51:02.796 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-04T20:51:02.796 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-04-04T20:51:02.797 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:51:02.797 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 7.4 M 2024-04-04T20:51:02.797 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-04T20:51:02.797 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-04T20:51:02.797 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-04T20:51:02.797 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-04T20:51:02.797 INFO:teuthology.orchestra.run.smithi106.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-04T20:51:02.797 INFO:teuthology.orchestra.run.smithi106.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-04T20:51:02.797 INFO:teuthology.orchestra.run.smithi106.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-04T20:51:02.797 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-04T20:51:02.797 INFO:teuthology.orchestra.run.smithi106.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-04T20:51:02.797 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-04T20:51:02.798 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-04T20:51:02.798 INFO:teuthology.orchestra.run.smithi106.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-04T20:51:02.798 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:02.798 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:51:02.798 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-04-04T20:51:02.798 INFO:teuthology.orchestra.run.smithi106.stdout:Install 12 Packages 2024-04-04T20:51:02.798 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:02.798 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 37 M 2024-04-04T20:51:02.799 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 209 M 2024-04-04T20:51:02.799 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:51:03.007 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-immutable-object-cache-19.0.0-1864.g66deef 648 kB/s | 145 kB 00:00 2024-04-04T20:51:03.008 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:03.008 INFO:teuthology.orchestra.run.smithi203.stdout:Total 644 kB/s | 145 kB 00:00 2024-04-04T20:51:03.008 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:51:03.015 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:51:03.015 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:51:03.068 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:51:03.068 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:51:03.207 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:51:03.266 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1864.g66deef15. 1/1 2024-04-04T20:51:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1864.g66deef15. 1/1 2024-04-04T20:51:03.291 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:51:03.291 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-04T20:51:03.291 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:03.316 INFO:teuthology.orchestra.run.smithi106.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 237 kB/s | 32 kB 00:00 2024-04-04T20:51:03.349 INFO:teuthology.orchestra.run.smithi106.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2024-04-04T20:51:03.391 INFO:teuthology.orchestra.run.smithi106.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 844 kB/s | 34 kB 00:00 2024-04-04T20:51:03.441 INFO:teuthology.orchestra.run.smithi106.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-04-04T20:51:03.617 INFO:teuthology.orchestra.run.smithi106.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-04-04T20:51:03.708 INFO:teuthology.orchestra.run.smithi106.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-04-04T20:51:03.734 INFO:teuthology.orchestra.run.smithi106.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2024-04-04T20:51:04.051 INFO:teuthology.orchestra.run.smithi106.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-04-04T20:51:04.184 INFO:teuthology.orchestra.run.smithi106.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-04-04T20:51:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1864.g66deef15. 1/1 2024-04-04T20:51:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:04.281 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:51:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:51:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:04.281 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:04.327 INFO:teuthology.orchestra.run.smithi106.stdout:(10/12): ceph-mgr-diskprediction-local-19.0.0-1 6.5 MB/s | 7.4 MB 00:01 2024-04-04T20:51:04.351 INFO:teuthology.orchestra.run.smithi106.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 651 kB/s | 195 kB 00:00 2024-04-04T20:51:04.377 INFO:teuthology.orchestra.run.smithi106.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-04-04T20:51:04.381 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:04.381 INFO:teuthology.orchestra.run.smithi106.stdout:Total 23 MB/s | 37 MB 00:01 2024-04-04T20:51:04.420 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:51:04.443 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:51:04.443 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:51:04.551 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr 2024-04-04T20:51:04.799 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:51:04.799 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:51:05.126 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:00 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:05.225 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-mgr-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:05.279 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:05.280 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-04T20:51:05.280 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:05.354 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-dashboard 2024-04-04T20:51:05.509 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:51:05.583 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-04T20:51:05.614 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-04T20:51:05.708 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-04T20:51:05.742 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-04T20:51:05.922 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-04T20:51:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-04T20:51:05.952 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:00 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:06.026 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-04T20:51:06.107 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:06.108 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-04-04T20:51:06.108 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-04T20:51:06.108 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-04-04T20:51:06.109 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:51:06.109 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 1.8 M 2024-04-04T20:51:06.109 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-04T20:51:06.109 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-04T20:51:06.109 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 31 k 2024-04-04T20:51:06.109 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 16 k 2024-04-04T20:51:06.109 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-04T20:51:06.109 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-04T20:51:06.109 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-04T20:51:06.109 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-04T20:51:06.109 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-04T20:51:06.109 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-04T20:51:06.109 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-04T20:51:06.110 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-04T20:51:06.110 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:06.110 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:51:06.110 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-04-04T20:51:06.110 INFO:teuthology.orchestra.run.smithi203.stdout:Install 12 Packages 2024-04-04T20:51:06.110 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:06.111 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 6.9 M 2024-04-04T20:51:06.111 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 26 M 2024-04-04T20:51:06.111 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:51:06.671 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-04T20:51:06.772 INFO:teuthology.orchestra.run.smithi203.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1864.g66d 67 kB/s | 16 kB 00:00 2024-04-04T20:51:06.778 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-04T20:51:06.797 INFO:teuthology.orchestra.run.smithi203.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1864.g66 116 kB/s | 31 kB 00:00 2024-04-04T20:51:06.922 INFO:teuthology.orchestra.run.smithi203.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1864.g66deef1 4.5 MB/s | 1.8 MB 00:00 2024-04-04T20:51:06.981 INFO:teuthology.orchestra.run.smithi203.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-04-04T20:51:07.006 INFO:teuthology.orchestra.run.smithi203.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 238 kB 00:00 2024-04-04T20:51:07.256 INFO:teuthology.orchestra.run.smithi203.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-04-04T20:51:07.290 INFO:teuthology.orchestra.run.smithi203.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.3 MB/s | 863 kB 00:00 2024-04-04T20:51:07.616 INFO:teuthology.orchestra.run.smithi203.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 864 kB/s | 548 kB 00:00 2024-04-04T20:51:07.716 INFO:teuthology.orchestra.run.smithi203.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 308 kB/s | 31 kB 00:00 2024-04-04T20:51:07.849 INFO:teuthology.orchestra.run.smithi203.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.4 MB/s | 188 kB 00:00 2024-04-04T20:51:07.875 INFO:teuthology.orchestra.run.smithi203.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 247 kB/s | 144 kB 00:00 2024-04-04T20:51:07.917 INFO:teuthology.orchestra.run.smithi203.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.1 MB/s | 2.0 MB 00:00 2024-04-04T20:51:07.919 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:07.919 INFO:teuthology.orchestra.run.smithi203.stdout:Total 3.8 MB/s | 6.9 MB 00:01 2024-04-04T20:51:07.937 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:51:07.954 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:51:07.954 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:51:08.006 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-04T20:51:08.124 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:51:08.125 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:51:08.642 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:51:08.718 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-04T20:51:08.753 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-04T20:51:08.799 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-04T20:51:08.830 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-04T20:51:08.936 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-04T20:51:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-04T20:51:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1864.g66dee 12/12 2024-04-04T20:51:09.065 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-04T20:51:09.159 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-04T20:51:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-04T20:51:09.239 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-04T20:51:09.279 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1864.g66deef15.el9 10/12 2024-04-04T20:51:09.533 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1864.g66deef15.el 11/12 2024-04-04T20:51:09.571 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1864.g66deef15.el9.noa 12/12 2024-04-04T20:51:10.373 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1864.g66deef15.el9.noa 12/12 2024-04-04T20:51:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1864.g66deef15.el 1/12 2024-04-04T20:51:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1864.g66deef15.el9.noa 2/12 2024-04-04T20:51:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1864.g66deef15.el9 3/12 2024-04-04T20:51:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-04T20:51:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-04T20:51:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-04T20:51:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-04T20:51:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-04T20:51:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-04T20:51:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-04T20:51:10.374 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-04T20:51:10.729 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-04T20:51:10.729 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:10.729 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:51:10.729 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-04T20:51:10.729 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:10.729 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:10.729 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:10.729 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-04T20:51:10.730 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-04T20:51:10.730 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-04T20:51:10.730 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-04T20:51:10.730 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-04T20:51:10.730 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-04T20:51:10.730 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-04T20:51:10.730 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-04T20:51:10.730 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:10.730 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:10.971 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-04T20:51:11.533 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:06 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:11.561 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1864.g66dee 12/12 2024-04-04T20:51:11.561 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1864.g66dee 1/12 2024-04-04T20:51:11.561 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-04T20:51:11.561 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-04T20:51:11.561 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-04T20:51:11.561 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-04T20:51:11.561 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-04T20:51:11.561 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-04T20:51:11.561 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-04T20:51:11.561 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-04T20:51:11.561 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-04T20:51:11.561 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-04T20:51:11.686 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:11.687 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-04-04T20:51:11.687 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-04T20:51:11.687 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-04-04T20:51:11.687 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:51:11.687 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 7.4 M 2024-04-04T20:51:11.687 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-04T20:51:11.687 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-04T20:51:11.687 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-04T20:51:11.687 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-04T20:51:11.687 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-04T20:51:11.687 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-04T20:51:11.687 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-04T20:51:11.688 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-04T20:51:11.688 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-04T20:51:11.688 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-04T20:51:11.688 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-04T20:51:11.688 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-04T20:51:11.688 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:11.688 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:51:11.688 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-04-04T20:51:11.688 INFO:teuthology.orchestra.run.smithi203.stdout:Install 12 Packages 2024-04-04T20:51:11.688 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:11.689 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 37 M 2024-04-04T20:51:11.689 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 209 M 2024-04-04T20:51:11.689 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:51:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-04T20:51:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:12.010 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:51:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-04T20:51:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-04T20:51:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-04T20:51:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-04T20:51:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-04T20:51:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-04T20:51:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-04T20:51:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-04T20:51:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-04T20:51:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-04T20:51:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-04T20:51:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:12.011 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:12.225 INFO:teuthology.orchestra.run.smithi203.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 218 kB/s | 32 kB 00:00 2024-04-04T20:51:12.276 INFO:teuthology.orchestra.run.smithi203.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 350 kB/s | 17 kB 00:00 2024-04-04T20:51:12.343 INFO:teuthology.orchestra.run.smithi203.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-04-04T20:51:12.368 INFO:teuthology.orchestra.run.smithi203.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 380 kB/s | 34 kB 00:00 2024-04-04T20:51:12.376 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr-rook 2024-04-04T20:51:12.585 INFO:teuthology.orchestra.run.smithi203.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-04-04T20:51:12.652 INFO:teuthology.orchestra.run.smithi203.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-04-04T20:51:12.677 INFO:teuthology.orchestra.run.smithi203.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-04-04T20:51:12.948 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:17 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:13.019 INFO:teuthology.orchestra.run.smithi203.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-04-04T20:51:13.102 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:13.103 INFO:teuthology.orchestra.run.smithi106.stdout:======================================================================================== 2024-04-04T20:51:13.103 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-04T20:51:13.104 INFO:teuthology.orchestra.run.smithi106.stdout:======================================================================================== 2024-04-04T20:51:13.104 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:51:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-rook noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 49 k 2024-04-04T20:51:13.104 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-04T20:51:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-04T20:51:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-04T20:51:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-04T20:51:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-04T20:51:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-04T20:51:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-04T20:51:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-04T20:51:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-04T20:51:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-04T20:51:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-04T20:51:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-04T20:51:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-04T20:51:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:13.105 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:51:13.105 INFO:teuthology.orchestra.run.smithi106.stdout:======================================================================================== 2024-04-04T20:51:13.105 INFO:teuthology.orchestra.run.smithi106.stdout:Install 13 Packages 2024-04-04T20:51:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:13.106 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 2.0 M 2024-04-04T20:51:13.106 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 26 M 2024-04-04T20:51:13.106 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:51:13.170 INFO:teuthology.orchestra.run.smithi203.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-04-04T20:51:13.195 INFO:teuthology.orchestra.run.smithi203.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-04-04T20:51:13.295 INFO:teuthology.orchestra.run.smithi203.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-04-04T20:51:13.566 INFO:teuthology.orchestra.run.smithi106.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 208 kB/s | 24 kB 00:00 2024-04-04T20:51:13.591 INFO:teuthology.orchestra.run.smithi106.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 127 kB/s | 18 kB 00:00 2024-04-04T20:51:13.633 INFO:teuthology.orchestra.run.smithi106.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-04-04T20:51:13.658 INFO:teuthology.orchestra.run.smithi106.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-04-04T20:51:13.683 INFO:teuthology.orchestra.run.smithi106.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-04T20:51:13.704 INFO:teuthology.orchestra.run.smithi203.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.6 MB/s | 7.4 MB 00:01 2024-04-04T20:51:13.706 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:13.706 INFO:teuthology.orchestra.run.smithi203.stdout:Total 18 MB/s | 37 MB 00:02 2024-04-04T20:51:13.709 INFO:teuthology.orchestra.run.smithi106.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-04-04T20:51:13.734 INFO:teuthology.orchestra.run.smithi106.stdout:(7/13): ceph-mgr-rook-19.0.0-1864.g66deef15.el9 173 kB/s | 49 kB 00:00 2024-04-04T20:51:13.746 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:51:13.770 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:51:13.771 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:51:13.834 INFO:teuthology.orchestra.run.smithi106.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 214 kB/s | 32 kB 00:00 2024-04-04T20:51:13.968 INFO:teuthology.orchestra.run.smithi106.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 866 kB/s | 203 kB 00:00 2024-04-04T20:51:14.035 INFO:teuthology.orchestra.run.smithi106.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 891 kB/s | 59 kB 00:00 2024-04-04T20:51:14.068 INFO:teuthology.orchestra.run.smithi106.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 4.4 MB/s | 1.0 MB 00:00 2024-04-04T20:51:14.094 INFO:teuthology.orchestra.run.smithi106.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-04-04T20:51:14.140 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:51:14.140 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:51:14.839 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:51:14.919 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-04T20:51:14.950 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-04T20:51:15.043 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-04T20:51:15.087 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-04T20:51:15.267 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-04T20:51:15.294 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-04T20:51:15.371 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-04T20:51:16.038 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-04T20:51:16.149 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-04T20:51:16.952 INFO:teuthology.orchestra.run.smithi106.stdout:(13/13): python3-certifi-2021.10.8-2.el9.noarch 4.6 kB/s | 15 kB 00:03 2024-04-04T20:51:16.955 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:16.955 INFO:teuthology.orchestra.run.smithi106.stdout:Total 544 kB/s | 2.0 MB 00:03 2024-04-04T20:51:16.955 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:51:16.970 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:51:16.970 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:51:17.065 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:51:17.066 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:51:17.401 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-04T20:51:17.596 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:51:17.756 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-04T20:51:17.817 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-04T20:51:17.875 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-04T20:51:17.921 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-04T20:51:17.962 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-04T20:51:18.057 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-04T20:51:18.174 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-04T20:51:18.222 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-04T20:51:18.424 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-04T20:51:18.455 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1864.g66dee 12/12 2024-04-04T20:51:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-04T20:51:18.993 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-04T20:51:19.027 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-04T20:51:19.067 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-04T20:51:19.097 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-rook-2:19.0.0-1864.g66deef15.el9.noarch 13/13 2024-04-04T20:51:19.909 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1864.g66deef15.el9.noarch 13/13 2024-04-04T20:51:19.909 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1864.g66deef15.el9.noarch 1/13 2024-04-04T20:51:19.909 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-04T20:51:19.909 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-04T20:51:19.909 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-04T20:51:19.909 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-04T20:51:19.909 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-04T20:51:19.909 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-04T20:51:19.909 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-04T20:51:19.910 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-04T20:51:19.910 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-04T20:51:19.910 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-04T20:51:19.910 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-04T20:51:20.274 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-04T20:51:20.275 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:20.275 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:51:20.275 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-rook-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:20.275 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-04T20:51:20.275 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-04T20:51:20.275 INFO:teuthology.orchestra.run.smithi106.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-04T20:51:20.275 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-04T20:51:20.275 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-04T20:51:20.275 INFO:teuthology.orchestra.run.smithi106.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-04T20:51:20.275 INFO:teuthology.orchestra.run.smithi106.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-04T20:51:20.275 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-04T20:51:20.275 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-04T20:51:20.276 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-04T20:51:20.276 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-04T20:51:20.276 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-04T20:51:20.276 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:20.276 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:20.522 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr-cephadm 2024-04-04T20:51:21.090 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:26 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:21.193 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1864.g66dee 12/12 2024-04-04T20:51:21.193 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1864.g66dee 1/12 2024-04-04T20:51:21.193 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-04T20:51:21.193 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-04T20:51:21.194 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-04T20:51:21.194 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-04T20:51:21.194 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-04T20:51:21.194 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-04T20:51:21.194 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-04T20:51:21.194 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-04T20:51:21.194 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-04T20:51:21.194 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-04T20:51:21.241 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:21.242 INFO:teuthology.orchestra.run.smithi106.stdout:======================================================================================== 2024-04-04T20:51:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-04T20:51:21.242 INFO:teuthology.orchestra.run.smithi106.stdout:======================================================================================== 2024-04-04T20:51:21.242 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:51:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 138 k 2024-04-04T20:51:21.242 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-04T20:51:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-04T20:51:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-04T20:51:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-04T20:51:21.242 INFO:teuthology.orchestra.run.smithi106.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-04T20:51:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-04T20:51:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:21.243 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:51:21.243 INFO:teuthology.orchestra.run.smithi106.stdout:======================================================================================== 2024-04-04T20:51:21.243 INFO:teuthology.orchestra.run.smithi106.stdout:Install 6 Packages 2024-04-04T20:51:21.243 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:21.243 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 6.8 M 2024-04-04T20:51:21.244 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 33 M 2024-04-04T20:51:21.244 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:51:21.638 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-04T20:51:21.638 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:21.638 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:51:21.638 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:21.638 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-04T20:51:21.638 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-04T20:51:21.638 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-04T20:51:21.639 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-04T20:51:21.639 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-04T20:51:21.639 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-04T20:51:21.639 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-04T20:51:21.639 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-04T20:51:21.639 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-04T20:51:21.639 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-04T20:51:21.639 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-04T20:51:21.639 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:21.639 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:21.903 INFO:teuthology.orchestra.run.smithi106.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-04-04T20:51:22.017 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-rook 2024-04-04T20:51:22.078 INFO:teuthology.orchestra.run.smithi106.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1864.g66deef15.e 240 kB/s | 138 kB 00:00 2024-04-04T20:51:22.103 INFO:teuthology.orchestra.run.smithi106.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 413 kB/s | 248 kB 00:00 2024-04-04T20:51:22.512 INFO:teuthology.orchestra.run.smithi106.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 133 kB/s | 58 kB 00:00 2024-04-04T20:51:22.537 INFO:teuthology.orchestra.run.smithi106.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 118 kB/s | 51 kB 00:00 2024-04-04T20:51:22.590 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:17 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:22.629 INFO:teuthology.orchestra.run.smithi106.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 755 kB/s | 548 kB 00:00 2024-04-04T20:51:22.631 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:22.632 INFO:teuthology.orchestra.run.smithi106.stdout:Total 4.9 MB/s | 6.8 MB 00:01 2024-04-04T20:51:22.645 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:51:22.655 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:51:22.655 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:51:22.742 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:22.743 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-04T20:51:22.743 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-04T20:51:22.743 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-04T20:51:22.743 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:51:22.743 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 49 k 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:22.744 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:51:22.745 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-04T20:51:22.745 INFO:teuthology.orchestra.run.smithi203.stdout:Install 13 Packages 2024-04-04T20:51:22.745 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:22.745 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:51:22.746 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:51:22.746 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 2.0 M 2024-04-04T20:51:22.746 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 26 M 2024-04-04T20:51:22.746 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:51:23.017 INFO:teuthology.orchestra.run.smithi203.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 194 kB/s | 24 kB 00:00 2024-04-04T20:51:23.042 INFO:teuthology.orchestra.run.smithi203.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 120 kB/s | 18 kB 00:00 2024-04-04T20:51:23.047 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:51:23.092 INFO:teuthology.orchestra.run.smithi203.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-04T20:51:23.118 INFO:teuthology.orchestra.run.smithi203.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-04T20:51:23.144 INFO:teuthology.orchestra.run.smithi203.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-04T20:51:23.169 INFO:teuthology.orchestra.run.smithi203.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-04-04T20:51:23.194 INFO:teuthology.orchestra.run.smithi203.stdout:(7/13): ceph-mgr-rook-19.0.0-1864.g66deef15.el9 163 kB/s | 49 kB 00:00 2024-04-04T20:51:23.209 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-04T20:51:23.255 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-04T20:51:23.361 INFO:teuthology.orchestra.run.smithi203.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 78 kB/s | 15 kB 00:00 2024-04-04T20:51:23.386 INFO:teuthology.orchestra.run.smithi203.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 133 kB/s | 32 kB 00:00 2024-04-04T20:51:23.578 INFO:teuthology.orchestra.run.smithi203.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-04-04T20:51:23.604 INFO:teuthology.orchestra.run.smithi203.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 495 kB/s | 203 kB 00:00 2024-04-04T20:51:23.712 INFO:teuthology.orchestra.run.smithi203.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 671 kB/s | 90 kB 00:00 2024-04-04T20:51:23.762 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-04T20:51:23.787 INFO:teuthology.orchestra.run.smithi203.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.4 MB/s | 1.0 MB 00:00 2024-04-04T20:51:23.789 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:23.790 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-04-04T20:51:23.790 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:51:23.805 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:51:23.806 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:51:23.841 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-04T20:51:23.903 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:51:23.903 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:51:23.903 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-04T20:51:23.926 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1864.g66deef15.el9.noarch 6/6 2024-04-04T20:51:24.426 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:51:24.593 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-04T20:51:24.647 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-04T20:51:24.702 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-04T20:51:24.718 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1864.g66deef15.el9.noarch 6/6 2024-04-04T20:51:24.718 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1864.g66deef15.el9.noarch 1/6 2024-04-04T20:51:24.718 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-04T20:51:24.718 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-04T20:51:24.718 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-04T20:51:24.718 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-04T20:51:24.740 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-04T20:51:24.781 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-04T20:51:24.883 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-04T20:51:25.001 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-04T20:51:25.028 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-04T20:51:25.029 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:25.029 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:51:25.029 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-cephadm-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:25.029 INFO:teuthology.orchestra.run.smithi106.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-04T20:51:25.029 INFO:teuthology.orchestra.run.smithi106.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-04T20:51:25.029 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-04T20:51:25.029 INFO:teuthology.orchestra.run.smithi106.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-04T20:51:25.029 INFO:teuthology.orchestra.run.smithi106.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-04T20:51:25.029 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:25.029 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:25.050 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-04T20:51:25.264 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-fuse 2024-04-04T20:51:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-04T20:51:25.827 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:30 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:25.831 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-04T20:51:25.865 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-04T20:51:25.913 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-04T20:51:25.935 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.0.0-1864.g66deef15.el9.noarch 13/13 2024-04-04T20:51:25.978 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:25.979 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:25.979 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-04T20:51:25.979 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:25.979 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:51:25.979 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse x86_64 2:19.0.0-1864.g66deef15.el9 ceph 887 k 2024-04-04T20:51:25.979 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-04T20:51:25.980 INFO:teuthology.orchestra.run.smithi106.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-04T20:51:25.980 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:25.980 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:51:25.980 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:25.980 INFO:teuthology.orchestra.run.smithi106.stdout:Install 2 Packages 2024-04-04T20:51:25.980 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:25.980 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 966 k 2024-04-04T20:51:25.981 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 2.8 M 2024-04-04T20:51:25.981 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:51:26.205 INFO:teuthology.orchestra.run.smithi106.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 351 kB/s | 79 kB 00:00 2024-04-04T20:51:26.690 INFO:teuthology.orchestra.run.smithi106.stdout:(2/2): ceph-fuse-19.0.0-1864.g66deef15.el9.x86_ 1.2 MB/s | 887 kB 00:00 2024-04-04T20:51:26.691 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:26.692 INFO:teuthology.orchestra.run.smithi106.stdout:Total 1.3 MB/s | 966 kB 00:00 2024-04-04T20:51:26.692 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:51:26.700 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1864.g66deef15.el9.noarch 13/13 2024-04-04T20:51:26.700 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1864.g66deef15.el9.noarch 1/13 2024-04-04T20:51:26.700 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-04T20:51:26.700 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-04T20:51:26.700 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-04T20:51:26.700 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-04T20:51:26.700 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-04T20:51:26.700 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-04T20:51:26.700 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-04T20:51:26.701 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-04T20:51:26.701 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-04T20:51:26.701 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-04T20:51:26.701 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-04T20:51:26.712 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:51:26.712 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:51:26.784 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:51:26.785 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:51:26.975 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:51:27.070 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-04T20:51:27.070 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:27.070 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:51:27.070 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:27.070 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-04T20:51:27.070 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-04T20:51:27.071 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-04T20:51:27.071 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-04T20:51:27.071 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-04T20:51:27.071 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-04T20:51:27.071 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-04T20:51:27.071 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-04T20:51:27.071 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-04T20:51:27.071 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-04T20:51:27.071 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-04T20:51:27.071 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-04T20:51:27.071 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:27.071 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:27.099 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-04T20:51:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 2/2 2024-04-04T20:51:27.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-cephadm 2024-04-04T20:51:27.882 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:22 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:27.902 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 2/2 2024-04-04T20:51:27.902 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 1/2 2024-04-04T20:51:28.037 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:28.038 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-04T20:51:28.038 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-04T20:51:28.038 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-04T20:51:28.038 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:51:28.039 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 138 k 2024-04-04T20:51:28.039 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-04T20:51:28.039 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-04T20:51:28.039 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-04T20:51:28.039 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-04T20:51:28.039 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-04T20:51:28.039 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-04T20:51:28.039 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:28.039 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:51:28.039 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-04T20:51:28.039 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-04-04T20:51:28.039 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:28.040 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 6.8 M 2024-04-04T20:51:28.040 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 33 M 2024-04-04T20:51:28.040 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:51:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-04T20:51:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:28.239 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:51:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-04T20:51:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:28.239 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:28.500 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-volume 2024-04-04T20:51:28.542 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-04-04T20:51:28.584 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1864.g66deef15.e 444 kB/s | 138 kB 00:00 2024-04-04T20:51:28.618 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 718 kB/s | 248 kB 00:00 2024-04-04T20:51:28.710 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 459 kB/s | 58 kB 00:00 2024-04-04T20:51:28.735 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 437 kB/s | 51 kB 00:00 2024-04-04T20:51:28.769 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.4 MB/s | 548 kB 00:00 2024-04-04T20:51:28.771 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:28.771 INFO:teuthology.orchestra.run.smithi203.stdout:Total 9.3 MB/s | 6.8 MB 00:00 2024-04-04T20:51:28.784 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:51:28.795 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:51:28.795 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:51:28.885 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:51:28.886 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:51:29.079 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:34 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:29.190 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:51:29.232 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:29.233 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:29.234 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-04T20:51:29.234 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:29.234 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:51:29.234 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-volume noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 264 k 2024-04-04T20:51:29.234 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-04T20:51:29.234 INFO:teuthology.orchestra.run.smithi106.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-04T20:51:29.234 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:29.234 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:51:29.234 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:29.234 INFO:teuthology.orchestra.run.smithi106.stdout:Install 2 Packages 2024-04-04T20:51:29.234 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:29.235 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 332 k 2024-04-04T20:51:29.235 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 1.5 M 2024-04-04T20:51:29.235 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:51:29.353 INFO:teuthology.orchestra.run.smithi106.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 582 kB/s | 69 kB 00:00 2024-04-04T20:51:29.367 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-04T20:51:29.421 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-04T20:51:29.628 INFO:teuthology.orchestra.run.smithi106.stdout:(2/2): ceph-volume-19.0.0-1864.g66deef15.el9.no 670 kB/s | 264 kB 00:00 2024-04-04T20:51:29.629 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:29.630 INFO:teuthology.orchestra.run.smithi106.stdout:Total 839 kB/s | 332 kB 00:00 2024-04-04T20:51:29.630 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:51:29.638 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:51:29.638 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:51:29.678 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:51:29.678 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:51:29.824 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:51:29.944 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-04T20:51:29.996 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-04T20:51:30.031 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-volume-2:19.0.0-1864.g66deef15.el9.noarch 2/2 2024-04-04T20:51:30.042 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-04T20:51:30.047 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-volume-2:19.0.0-1864.g66deef15.el9.noarch 2/2 2024-04-04T20:51:30.047 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:51:30.047 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-04T20:51:30.047 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-04T20:51:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1864.g66deef15.el9.noarch 6/6 2024-04-04T20:51:30.912 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-volume-2:19.0.0-1864.g66deef15.el9.noarch 1/2 2024-04-04T20:51:30.960 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1864.g66deef15.el9.noarch 6/6 2024-04-04T20:51:30.960 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1864.g66deef15.el9.noarch 1/6 2024-04-04T20:51:30.960 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-04T20:51:30.961 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-04T20:51:30.961 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-04T20:51:30.961 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-04T20:51:31.271 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-04T20:51:31.271 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:31.271 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:51:31.271 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-volume-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:31.271 INFO:teuthology.orchestra.run.smithi106.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-04T20:51:31.271 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:31.271 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:31.274 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-04T20:51:31.274 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:31.274 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:51:31.274 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:31.274 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-04T20:51:31.274 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-04T20:51:31.274 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-04T20:51:31.274 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-04T20:51:31.274 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-04T20:51:31.274 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:31.274 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:31.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-fuse 2024-04-04T20:51:31.627 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librados-devel 2024-04-04T20:51:32.083 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:27 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:32.208 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:37 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:32.237 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:32.237 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:32.238 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-04T20:51:32.238 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:32.238 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:51:32.238 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.0.0-1864.g66deef15.el9 ceph 887 k 2024-04-04T20:51:32.238 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-04T20:51:32.238 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-04T20:51:32.238 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:32.238 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:51:32.238 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:32.238 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-04-04T20:51:32.238 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:32.239 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 966 k 2024-04-04T20:51:32.239 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.8 M 2024-04-04T20:51:32.239 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:51:32.355 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 679 kB/s | 79 kB 00:00 2024-04-04T20:51:32.360 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:32.362 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:32.362 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:51:32.362 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:32.362 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:51:32.362 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel x86_64 2:19.0.0-1864.g66deef15.el9 ceph 127 k 2024-04-04T20:51:32.362 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:32.362 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:51:32.362 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:32.362 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:51:32.362 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:32.362 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 127 k 2024-04-04T20:51:32.362 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 456 k 2024-04-04T20:51:32.363 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:51:32.555 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-fuse-19.0.0-1864.g66deef15.el9.x86_ 2.7 MB/s | 887 kB 00:00 2024-04-04T20:51:32.556 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:32.557 INFO:teuthology.orchestra.run.smithi203.stdout:Total 3.0 MB/s | 966 kB 00:00 2024-04-04T20:51:32.557 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:51:32.576 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:51:32.577 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:51:32.649 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:51:32.649 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:51:32.667 INFO:teuthology.orchestra.run.smithi106.stdout:librados-devel-19.0.0-1864.g66deef15.el9.x86_64 417 kB/s | 127 kB 00:00 2024-04-04T20:51:32.667 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:32.667 INFO:teuthology.orchestra.run.smithi106.stdout:Total 415 kB/s | 127 kB 00:00 2024-04-04T20:51:32.668 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:51:32.673 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:51:32.673 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:51:32.730 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:51:32.730 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:51:32.839 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:51:32.889 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:51:32.973 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-04T20:51:32.982 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librados-devel-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:33.046 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 2/2 2024-04-04T20:51:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados-devel-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 2/2 2024-04-04T20:51:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 1/2 2024-04-04T20:51:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados-devel-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:33.778 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:51:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:51:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:33.779 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:33.992 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-04T20:51:33.992 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:33.992 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:51:33.992 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-04T20:51:33.992 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:33.992 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:34.002 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install libcephfs2 2024-04-04T20:51:34.300 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-volume 2024-04-04T20:51:34.576 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:39 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:34.676 INFO:teuthology.orchestra.run.smithi106.stdout:Package libcephfs2-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:34.732 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:34.733 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-04T20:51:34.733 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:34.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install libcephfs-devel 2024-04-04T20:51:34.882 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:29 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:35.037 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:35.038 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:35.038 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-04T20:51:35.038 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:35.038 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:51:35.038 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.0.0-1864.g66deef15.el9 ceph-noarch 264 k 2024-04-04T20:51:35.038 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-04T20:51:35.038 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-04T20:51:35.038 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:35.038 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:51:35.039 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:35.039 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-04-04T20:51:35.039 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:35.039 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 332 k 2024-04-04T20:51:35.039 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.5 M 2024-04-04T20:51:35.039 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:51:35.170 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 526 kB/s | 69 kB 00:00 2024-04-04T20:51:35.321 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-volume-19.0.0-1864.g66deef15.el9.no 939 kB/s | 264 kB 00:00 2024-04-04T20:51:35.322 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:35.322 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-04-04T20:51:35.322 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:51:35.330 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:51:35.330 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:51:35.356 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:40 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:35.365 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:51:35.366 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:51:35.508 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:35.509 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:35.509 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:51:35.509 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:35.509 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:51:35.509 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel x86_64 2:19.0.0-1864.g66deef15.el9 ceph 31 k 2024-04-04T20:51:35.509 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:35.509 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:51:35.509 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:35.509 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:51:35.509 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:35.510 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 31 k 2024-04-04T20:51:35.510 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 140 k 2024-04-04T20:51:35.510 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:51:35.516 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:51:35.704 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-04T20:51:35.740 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.0.0-1864.g66deef15.el9.noarch 2/2 2024-04-04T20:51:35.745 INFO:teuthology.orchestra.run.smithi106.stdout:libcephfs-devel-19.0.0-1864.g66deef15.el9.x86_6 133 kB/s | 31 kB 00:00 2024-04-04T20:51:35.745 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:35.745 INFO:teuthology.orchestra.run.smithi106.stdout:Total 132 kB/s | 31 kB 00:00 2024-04-04T20:51:35.746 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:51:35.749 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:51:35.749 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:51:35.756 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.0.0-1864.g66deef15.el9.noarch 2/2 2024-04-04T20:51:35.756 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:51:35.756 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-04T20:51:35.756 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:35.775 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:51:35.775 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:51:35.898 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:51:35.992 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs-devel-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:36.572 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.0.0-1864.g66deef15.el9.noarch 1/2 2024-04-04T20:51:36.634 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs-devel-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:36.634 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:36.635 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:51:36.635 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:51:36.635 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:36.635 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:36.783 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librados2 2024-04-04T20:51:36.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-04T20:51:36.931 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:36.931 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:51:36.932 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.0.0-1864.g66deef15.el9.noarch 2024-04-04T20:51:36.932 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-04T20:51:36.932 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:36.932 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:37.262 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados-devel 2024-04-04T20:51:37.346 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:42 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:37.445 INFO:teuthology.orchestra.run.smithi106.stdout:Package librados2-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:37.498 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:37.498 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-04T20:51:37.498 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:37.551 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librbd1 2024-04-04T20:51:37.842 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:32 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:37.999 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:37.999 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:37.999 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:51:38.000 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:38.000 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:51:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.0.0-1864.g66deef15.el9 ceph 127 k 2024-04-04T20:51:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:38.000 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:51:38.000 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:38.000 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:51:38.000 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:38.000 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 127 k 2024-04-04T20:51:38.000 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 456 k 2024-04-04T20:51:38.000 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:51:38.111 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:43 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:38.210 INFO:teuthology.orchestra.run.smithi106.stdout:Package librbd1-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:38.263 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:38.264 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-04T20:51:38.264 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:38.312 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-rados 2024-04-04T20:51:38.421 INFO:teuthology.orchestra.run.smithi203.stdout:librados-devel-19.0.0-1864.g66deef15.el9.x86_64 301 kB/s | 127 kB 00:00 2024-04-04T20:51:38.421 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:38.421 INFO:teuthology.orchestra.run.smithi203.stdout:Total 301 kB/s | 127 kB 00:00 2024-04-04T20:51:38.422 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:51:38.427 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:51:38.427 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:51:38.484 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:51:38.484 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:51:38.652 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:51:38.745 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:38.874 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:43 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:38.972 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-rados-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:39.024 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:39.025 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-04T20:51:39.025 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:39.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-rgw 2024-04-04T20:51:39.228 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados-devel-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:39.533 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:39.533 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:39.533 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:51:39.533 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:51:39.533 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:39.534 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:39.638 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:44 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:39.737 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-rgw-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:39.788 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:39.789 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs2 2024-04-04T20:51:39.790 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-04T20:51:39.790 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:39.837 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-cephfs 2024-04-04T20:51:40.373 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:35 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:40.434 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:45 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:40.477 INFO:teuthology.orchestra.run.smithi203.stdout:Package libcephfs2-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:40.533 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:40.534 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-04T20:51:40.534 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:40.537 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-cephfs-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:40.591 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:40.592 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-04T20:51:40.592 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:40.616 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs-devel 2024-04-04T20:51:40.643 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-rbd 2024-04-04T20:51:41.184 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:36 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:41.219 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:46 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:41.318 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-rbd-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:41.338 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:41.339 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:41.339 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:51:41.339 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:41.339 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:51:41.339 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.0.0-1864.g66deef15.el9 ceph 31 k 2024-04-04T20:51:41.339 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:41.339 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:51:41.339 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:41.340 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:51:41.340 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:41.340 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 31 k 2024-04-04T20:51:41.340 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 140 k 2024-04-04T20:51:41.340 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:51:41.371 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:41.372 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-04T20:51:41.372 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:41.419 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-fuse 2024-04-04T20:51:41.509 INFO:teuthology.orchestra.run.smithi203.stdout:libcephfs-devel-19.0.0-1864.g66deef15.el9.x86_6 184 kB/s | 31 kB 00:00 2024-04-04T20:51:41.509 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:41.510 INFO:teuthology.orchestra.run.smithi203.stdout:Total 183 kB/s | 31 kB 00:00 2024-04-04T20:51:41.510 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:51:41.513 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:51:41.513 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:51:41.534 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:51:41.534 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:51:41.665 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:51:41.758 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:41.980 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:46 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:42.131 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:42.132 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:42.132 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-04-04T20:51:42.132 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:42.132 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:51:42.132 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse x86_64 2:19.0.0-1864.g66deef15.el9 ceph 85 k 2024-04-04T20:51:42.132 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:42.132 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:51:42.132 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:42.132 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:51:42.132 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:42.132 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 85 k 2024-04-04T20:51:42.132 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 231 k 2024-04-04T20:51:42.132 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:51:42.137 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:42.423 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:42.423 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:42.423 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:51:42.423 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:51:42.423 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:42.424 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:42.448 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-fuse-19.0.0-1864.g66deef15.el9.x86_64.rpm 270 kB/s | 85 kB 00:00 2024-04-04T20:51:42.449 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:42.449 INFO:teuthology.orchestra.run.smithi106.stdout:Total 269 kB/s | 85 kB 00:00 2024-04-04T20:51:42.449 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:51:42.456 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:51:42.456 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:51:42.512 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:51:42.512 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:51:42.587 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados2 2024-04-04T20:51:42.657 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:51:42.746 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:43.148 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:43.157 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:38 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:43.257 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:43.310 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:43.311 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-04T20:51:43.311 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:43.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librbd1 2024-04-04T20:51:43.437 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:43.437 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:43.438 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:51:43.438 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:51:43.438 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:43.438 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:43.667 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-mirror 2024-04-04T20:51:43.932 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:38 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:44.033 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:44.086 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:44.087 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-04T20:51:44.087 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:44.141 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rados 2024-04-04T20:51:44.243 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:49 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:44.397 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:44.397 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:44.397 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:51:44.398 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:44.398 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:51:44.398 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror x86_64 2:19.0.0-1864.g66deef15.el9 ceph 3.1 M 2024-04-04T20:51:44.398 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:44.398 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:51:44.398 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:44.398 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:51:44.398 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:44.398 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 3.1 M 2024-04-04T20:51:44.398 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 13 M 2024-04-04T20:51:44.398 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:51:44.718 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:39 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:44.816 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rados-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:44.868 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:44.869 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-04T20:51:44.869 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:44.925 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rgw 2024-04-04T20:51:45.343 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-mirror-19.0.0-1864.g66deef15.el9.x86_64.rpm 3.3 MB/s | 3.1 MB 00:00 2024-04-04T20:51:45.343 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:45.344 INFO:teuthology.orchestra.run.smithi106.stdout:Total 3.3 MB/s | 3.1 MB 00:00 2024-04-04T20:51:45.344 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:51:45.355 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:51:45.355 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:51:45.433 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:51:45.433 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:51:45.494 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:40 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:45.593 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rgw-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:45.645 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:45.646 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-04T20:51:45.646 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:45.709 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-cephfs 2024-04-04T20:51:45.851 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:51:45.902 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-mirror-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:45.926 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:45.927 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:51:45.927 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-04T20:51:45.927 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-04T20:51:45.927 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-04T20:51:45.927 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:46.277 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:41 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:46.377 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-cephfs-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:46.429 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:46.430 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-04T20:51:46.430 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:46.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rbd 2024-04-04T20:51:47.035 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-mirror-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:47.035 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:47.036 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:51:47.036 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:51:47.036 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:47.036 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:47.066 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:42 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:47.170 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rbd-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:51:47.226 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:47.227 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-04T20:51:47.227 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:47.279 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-nbd 2024-04-04T20:51:47.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-fuse 2024-04-04T20:51:47.848 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:42 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:47.856 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:52 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:47.999 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:48.000 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:48.000 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-04-04T20:51:48.000 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:48.000 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:51:48.000 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.0.0-1864.g66deef15.el9 ceph 85 k 2024-04-04T20:51:48.001 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:48.001 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:51:48.001 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:48.001 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:51:48.001 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:48.001 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 85 k 2024-04-04T20:51:48.001 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 231 k 2024-04-04T20:51:48.001 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:51:48.008 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:48.008 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:48.009 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-04-04T20:51:48.009 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:48.009 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:51:48.009 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd x86_64 2:19.0.0-1864.g66deef15.el9 ceph 170 k 2024-04-04T20:51:48.009 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:48.009 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:51:48.009 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:48.009 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:51:48.009 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:48.010 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 170 k 2024-04-04T20:51:48.010 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 490 k 2024-04-04T20:51:48.010 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:51:48.246 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-nbd-19.0.0-1864.g66deef15.el9.x86_64.rpm 724 kB/s | 170 kB 00:00 2024-04-04T20:51:48.246 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:48.247 INFO:teuthology.orchestra.run.smithi106.stdout:Total 719 kB/s | 170 kB 00:00 2024-04-04T20:51:48.247 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:51:48.255 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:51:48.255 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:51:48.296 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-fuse-19.0.0-1864.g66deef15.el9.x86_64.rpm 289 kB/s | 85 kB 00:00 2024-04-04T20:51:48.296 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:48.297 INFO:teuthology.orchestra.run.smithi203.stdout:Total 288 kB/s | 85 kB 00:00 2024-04-04T20:51:48.297 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:51:48.304 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:51:48.304 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:51:48.311 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:51:48.311 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:51:48.359 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:51:48.360 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:51:48.466 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:51:48.501 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:51:48.577 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-nbd-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:48.612 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:48.990 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:49.015 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:49.260 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:49.260 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:49.260 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:51:49.261 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:51:49.261 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:49.261 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:49.293 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-nbd-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:49.293 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:49.294 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:51:49.294 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:51:49.294 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:49.294 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:51:49.521 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-base-debuginfo 2024-04-04T20:51:49.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-mirror 2024-04-04T20:51:50.099 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:55 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:51:50.122 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:45 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:50.255 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:51:50.255 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:50.255 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:51:50.255 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:50.256 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:51:50.256 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 68 M 2024-04-04T20:51:50.256 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-04T20:51:50.256 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 23 M 2024-04-04T20:51:50.256 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:50.256 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:51:50.256 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:51:50.256 INFO:teuthology.orchestra.run.smithi106.stdout:Install 2 Packages 2024-04-04T20:51:50.256 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:51:50.257 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 91 M 2024-04-04T20:51:50.257 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 501 M 2024-04-04T20:51:50.257 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:51:50.281 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:50.282 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:50.282 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:51:50.282 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:50.282 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:51:50.282 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.0.0-1864.g66deef15.el9 ceph 3.1 M 2024-04-04T20:51:50.282 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:50.282 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:51:50.282 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:50.282 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:51:50.282 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:50.283 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 3.1 M 2024-04-04T20:51:50.283 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 13 M 2024-04-04T20:51:50.283 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:51:52.042 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-mirror-19.0.0-1864.g66deef15.el9.x86_64.rpm 1.8 MB/s | 3.1 MB 00:01 2024-04-04T20:51:52.042 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:52.043 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.8 MB/s | 3.1 MB 00:01 2024-04-04T20:51:52.043 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:51:52.053 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:51:52.054 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:51:52.131 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:51:52.132 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:51:52.490 INFO:teuthology.orchestra.run.smithi106.stdout:(1/2): ceph-debuginfo-19.0.0-1864.g66deef15.el9 10 MB/s | 23 MB 00:02 2024-04-04T20:51:52.564 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:51:52.626 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:52.651 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:52.651 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-04T20:51:52.651 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-04T20:51:52.651 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-04T20:51:52.651 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-04T20:51:52.651 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:53.830 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:53.830 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:53.830 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:51:53.830 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:51:53.830 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:53.830 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:54.128 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-nbd 2024-04-04T20:51:54.465 INFO:teuthology.orchestra.run.smithi106.stdout:(2/2): ceph-base-debuginfo-19.0.0-1864.g66deef1 16 MB/s | 68 MB 00:04 2024-04-04T20:51:54.467 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:54.467 INFO:teuthology.orchestra.run.smithi106.stdout:Total 22 MB/s | 91 MB 00:04 2024-04-04T20:51:54.467 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:51:54.472 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:51:54.472 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:51:54.706 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:49 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:54.857 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:54.858 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:54.858 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-04-04T20:51:54.858 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:54.858 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:51:54.858 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.0.0-1864.g66deef15.el9 ceph 170 k 2024-04-04T20:51:54.858 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:54.858 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:51:54.858 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:54.859 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:51:54.859 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:54.859 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 170 k 2024-04-04T20:51:54.859 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 490 k 2024-04-04T20:51:54.859 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:51:55.066 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:51:55.066 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:51:55.413 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-nbd-19.0.0-1864.g66deef15.el9.x86_64.rpm 308 kB/s | 170 kB 00:00 2024-04-04T20:51:55.414 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:51:55.414 INFO:teuthology.orchestra.run.smithi203.stdout:Total 307 kB/s | 170 kB 00:00 2024-04-04T20:51:55.414 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:51:55.422 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:51:55.422 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:51:55.478 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:51:55.479 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:51:55.630 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:51:55.737 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:56.194 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:56.471 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:51:56.471 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:56.472 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:51:56.472 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:51:56.472 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:56.472 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:51:56.734 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base-debuginfo 2024-04-04T20:51:57.308 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:52 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:51:57.464 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:51:57.465 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:57.465 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:51:57.465 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:57.465 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:51:57.465 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 68 M 2024-04-04T20:51:57.465 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-04T20:51:57.465 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 23 M 2024-04-04T20:51:57.465 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:57.465 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:51:57.465 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:51:57.465 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-04-04T20:51:57.465 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:51:57.466 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 91 M 2024-04-04T20:51:57.466 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 501 M 2024-04-04T20:51:57.466 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:51:58.218 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:51:59.734 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): ceph-debuginfo-19.0.0-1864.g66deef15.el9 10 MB/s | 23 MB 00:02 2024-04-04T20:52:02.184 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-base-debuginfo-19.0.0-1864.g66deef1 15 MB/s | 68 MB 00:04 2024-04-04T20:52:02.186 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:52:02.186 INFO:teuthology.orchestra.run.smithi203.stdout:Total 19 MB/s | 91 MB 00:04 2024-04-04T20:52:02.187 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:52:02.192 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:52:02.192 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:52:02.785 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:52:02.785 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:52:05.405 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 1/2 2024-04-04T20:52:05.482 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 2/2 2024-04-04T20:52:05.972 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:52:08.775 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 2/2 2024-04-04T20:52:08.775 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/2 2024-04-04T20:52:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2/2 2024-04-04T20:52:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:52:09.055 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:52:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:52:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:52:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:52:09.056 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:52:09.247 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-common-debuginfo 2024-04-04T20:52:09.822 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:14 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:52:09.984 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:52:09.984 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:52:09.984 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:52:09.984 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:52:09.985 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:52:09.985 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 422 M 2024-04-04T20:52:09.985 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:52:09.985 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:52:09.985 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:52:09.985 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:52:09.985 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:52:09.985 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 422 M 2024-04-04T20:52:09.985 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 1.7 G 2024-04-04T20:52:09.985 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:52:13.125 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 1/2 2024-04-04T20:52:13.210 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 2/2 2024-04-04T20:52:16.742 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 2/2 2024-04-04T20:52:16.742 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/2 2024-04-04T20:52:17.010 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2/2 2024-04-04T20:52:17.010 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:52:17.010 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:52:17.010 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:52:17.010 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:52:17.010 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:52:17.010 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:52:17.252 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-common-debuginfo 2024-04-04T20:52:17.826 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:12 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:52:17.988 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:52:17.989 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:52:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:52:17.989 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:52:17.989 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:52:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 422 M 2024-04-04T20:52:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:52:17.989 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:52:17.989 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:52:17.990 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:52:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:52:17.990 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 422 M 2024-04-04T20:52:17.990 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.7 G 2024-04-04T20:52:17.990 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:53:28.847 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-common-debuginfo-19.0.0-1864.g66deef15.el9 5.3 MB/s | 422 MB 01:18 2024-04-04T20:53:28.847 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:53:28.847 INFO:teuthology.orchestra.run.smithi106.stdout:Total 5.3 MB/s | 422 MB 01:18 2024-04-04T20:53:28.847 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:53:28.851 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:53:28.851 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:53:31.354 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:53:31.354 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:53:53.629 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-common-debuginfo-19.0.0-1864.g66deef15.el9 4.4 MB/s | 422 MB 01:35 2024-04-04T20:53:53.751 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:53:53.751 INFO:teuthology.orchestra.run.smithi203.stdout:Total 4.4 MB/s | 422 MB 01:35 2024-04-04T20:53:53.751 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:53:53.751 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:53:53.752 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:53:56.148 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:53:56.148 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:54:13.960 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:54:14.048 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T20:54:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T20:54:19.264 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T20:54:19.265 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:54:19.265 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:54:19.265 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:54:19.265 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:54:19.265 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:54:19.495 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-04-04T20:54:20.060 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:04:25 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:54:20.210 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:54:20.211 INFO:teuthology.orchestra.run.smithi106.stdout:======================================================================================== 2024-04-04T20:54:20.211 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:54:20.211 INFO:teuthology.orchestra.run.smithi106.stdout:======================================================================================== 2024-04-04T20:54:20.211 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:54:20.211 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 1.4 M 2024-04-04T20:54:20.211 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:54:20.211 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:54:20.211 INFO:teuthology.orchestra.run.smithi106.stdout:======================================================================================== 2024-04-04T20:54:20.211 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:54:20.212 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:54:20.212 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 1.4 M 2024-04-04T20:54:20.212 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 6.5 M 2024-04-04T20:54:20.212 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:54:20.920 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-18 2.0 MB/s | 1.4 MB 00:00 2024-04-04T20:54:20.920 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:54:20.921 INFO:teuthology.orchestra.run.smithi106.stdout:Total 2.0 MB/s | 1.4 MB 00:00 2024-04-04T20:54:20.921 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:54:20.924 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:54:20.924 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:54:20.965 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:54:20.965 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:54:21.223 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:54:21.336 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1864. 1/1 2024-04-04T20:54:21.753 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1864. 1/1 2024-04-04T20:54:22.007 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1864. 1/1 2024-04-04T20:54:22.007 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:54:22.007 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:54:22.007 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:54:22.007 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:54:22.007 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:54:22.167 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-radosgw-debuginfo 2024-04-04T20:54:22.735 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:04:27 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:54:22.885 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:54:22.886 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:54:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:54:22.886 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:54:22.886 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:54:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 176 M 2024-04-04T20:54:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:54:22.886 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:54:22.886 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:54:22.886 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:54:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:54:22.887 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 176 M 2024-04-04T20:54:22.887 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 757 M 2024-04-04T20:54:22.887 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:54:32.730 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-radosgw-debuginfo-19.0.0-1864.g66deef15.el 18 MB/s | 176 MB 00:09 2024-04-04T20:54:32.731 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:54:32.732 INFO:teuthology.orchestra.run.smithi106.stdout:Total 18 MB/s | 176 MB 00:09 2024-04-04T20:54:32.732 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:54:32.736 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:54:32.736 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:54:33.809 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:54:33.809 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:54:39.327 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:54:39.567 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T20:54:48.236 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T20:54:48.497 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T20:54:48.497 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:54:48.497 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:54:48.497 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:54:48.497 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:54:48.497 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:54:48.754 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-04-04T20:54:49.351 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:04:44 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:54:49.505 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:54:49.506 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-04T20:54:49.506 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:54:49.506 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-04T20:54:49.506 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:54:49.506 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 1.4 M 2024-04-04T20:54:49.506 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:54:49.507 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:54:49.507 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-04T20:54:49.507 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:54:49.507 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:54:49.507 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 1.4 M 2024-04-04T20:54:49.507 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 6.5 M 2024-04-04T20:54:49.507 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:54:49.818 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-18 4.6 MB/s | 1.4 MB 00:00 2024-04-04T20:54:49.819 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:54:49.819 INFO:teuthology.orchestra.run.smithi203.stdout:Total 4.6 MB/s | 1.4 MB 00:00 2024-04-04T20:54:49.819 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:54:49.823 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:54:49.823 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:54:49.865 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:54:49.865 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:54:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:54:50.235 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1864. 1/1 2024-04-04T20:54:50.617 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1864. 1/1 2024-04-04T20:54:50.898 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1864. 1/1 2024-04-04T20:54:50.898 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:54:50.898 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:54:50.898 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:54:50.898 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:54:50.898 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:54:51.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw-debuginfo 2024-04-04T20:54:51.663 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:04:46 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:54:51.816 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:54:51.817 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:54:51.817 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:54:51.817 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:54:51.817 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:54:51.817 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 176 M 2024-04-04T20:54:51.817 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:54:51.817 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:54:51.818 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:54:51.818 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:54:51.818 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:54:51.818 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 176 M 2024-04-04T20:54:51.818 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 757 M 2024-04-04T20:54:51.818 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:54:52.279 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:54:52.502 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x 1/1 2024-04-04T20:54:57.143 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x 1/1 2024-04-04T20:54:57.392 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x 1/1 2024-04-04T20:54:57.392 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:54:57.392 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:54:57.393 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:54:57.393 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:54:57.393 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:54:57.569 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-test-debuginfo 2024-04-04T20:54:58.132 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:05:03 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:54:58.282 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:54:58.283 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:54:58.283 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:54:58.283 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:54:58.283 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:54:58.283 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 675 M 2024-04-04T20:54:58.283 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:54:58.284 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:54:58.284 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:54:58.284 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:54:58.284 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:54:58.284 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 675 M 2024-04-04T20:54:58.284 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 3.1 G 2024-04-04T20:54:58.284 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:55:04.296 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-radosgw-debuginfo-19.0.0-1864.g66deef15.el 14 MB/s | 176 MB 00:12 2024-04-04T20:55:04.296 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:55:04.297 INFO:teuthology.orchestra.run.smithi203.stdout:Total 14 MB/s | 176 MB 00:12 2024-04-04T20:55:04.297 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:55:04.301 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:55:04.301 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:55:05.371 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:55:05.371 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:55:23.533 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:55:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x 1/1 2024-04-04T20:55:28.720 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x 1/1 2024-04-04T20:55:28.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x 1/1 2024-04-04T20:55:28.976 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:55:28.977 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:55:28.977 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:55:28.977 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:55:28.977 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:55:29.199 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-test-debuginfo 2024-04-04T20:55:29.767 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:05:24 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:55:29.916 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:55:29.917 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:55:29.917 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:55:29.917 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:55:29.917 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:55:29.917 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 675 M 2024-04-04T20:55:29.917 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:55:29.917 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:55:29.917 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:55:29.917 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:55:29.917 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:55:29.918 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 675 M 2024-04-04T20:55:29.918 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 3.1 G 2024-04-04T20:55:29.918 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:56:08.684 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-test-debuginfo-19.0.0-1864.g66deef15.el9.x 17 MB/s | 675 MB 00:38 2024-04-04T20:56:08.685 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:56:08.685 INFO:teuthology.orchestra.run.smithi203.stdout:Total 17 MB/s | 675 MB 00:38 2024-04-04T20:56:08.685 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:56:08.691 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:56:08.691 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:56:12.671 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:56:12.672 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:56:31.530 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-test-debuginfo-19.0.0-1864.g66deef15.el9.x 7.2 MB/s | 675 MB 01:33 2024-04-04T20:56:31.531 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:56:31.531 INFO:teuthology.orchestra.run.smithi106.stdout:Total 7.2 MB/s | 675 MB 01:33 2024-04-04T20:56:31.531 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:56:31.537 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:56:31.537 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:56:35.510 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:56:35.511 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:57:23.902 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:57:24.010 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:57:33.897 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:57:34.159 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:57:34.159 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:57:34.159 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:57:34.159 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:57:34.159 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:57:34.159 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:57:34.458 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base-debuginfo 2024-04-04T20:57:35.023 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:07:30 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:57:35.124 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-base-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:57:35.176 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:57:35.177 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-04T20:57:35.177 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:57:35.248 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-debuginfo 2024-04-04T20:57:35.817 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:07:30 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:57:35.966 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:57:35.967 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:57:35.967 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:57:35.967 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:57:35.967 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:57:35.967 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 20 M 2024-04-04T20:57:35.967 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:57:35.967 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:57:35.968 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:57:35.968 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:57:35.968 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:57:35.968 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 20 M 2024-04-04T20:57:35.968 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 104 M 2024-04-04T20:57:35.968 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:57:37.114 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-mgr-debuginfo-19.0.0-1864.g66deef15.el9.x8 17 MB/s | 20 MB 00:01 2024-04-04T20:57:37.114 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:57:37.115 INFO:teuthology.orchestra.run.smithi203.stdout:Total 17 MB/s | 20 MB 00:01 2024-04-04T20:57:37.115 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:57:37.118 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:57:37.118 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:57:37.269 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:57:37.269 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:57:39.538 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:57:39.712 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:57:40.763 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:57:41.024 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:57:41.024 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:57:41.024 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:57:41.024 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:57:41.024 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:57:41.024 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:57:41.211 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mds-debuginfo 2024-04-04T20:57:41.776 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:07:36 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:57:41.926 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:57:41.926 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:57:41.926 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:57:41.926 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:57:41.926 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:57:41.927 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 49 M 2024-04-04T20:57:41.927 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:57:41.927 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:57:41.927 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:57:41.927 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:57:41.927 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:57:41.928 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 49 M 2024-04-04T20:57:41.928 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 177 M 2024-04-04T20:57:41.928 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:57:44.341 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-mds-debuginfo-19.0.0-1864.g66deef15.el9.x8 20 MB/s | 49 MB 00:02 2024-04-04T20:57:44.342 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:57:44.343 INFO:teuthology.orchestra.run.smithi203.stdout:Total 20 MB/s | 49 MB 00:02 2024-04-04T20:57:44.343 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:57:44.347 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:57:44.347 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:57:44.669 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:57:44.669 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:57:46.530 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:57:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:57:49.490 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:57:49.597 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:57:51.115 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:57:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:57:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:57:51.379 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:57:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:57:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:57:51.379 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:57:51.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mon-debuginfo 2024-04-04T20:57:52.137 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:07:47 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:57:52.288 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:57:52.289 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:57:52.289 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:57:52.289 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:57:52.289 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:57:52.289 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 99 M 2024-04-04T20:57:52.289 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:57:52.289 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:57:52.289 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:57:52.289 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:57:52.289 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:57:52.289 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 99 M 2024-04-04T20:57:52.289 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 450 M 2024-04-04T20:57:52.290 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:57:56.065 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:57:56.318 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:57:56.318 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:57:56.318 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:57:56.318 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:57:56.319 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:57:56.319 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:57:56.593 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-base-debuginfo 2024-04-04T20:57:57.079 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-mon-debuginfo-19.0.0-1864.g66deef15.el9.x8 21 MB/s | 99 MB 00:04 2024-04-04T20:57:57.080 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:57:57.081 INFO:teuthology.orchestra.run.smithi203.stdout:Total 21 MB/s | 99 MB 00:04 2024-04-04T20:57:57.081 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:57:57.085 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:57:57.085 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:57:57.156 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:08:02 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:57:57.255 INFO:teuthology.orchestra.run.smithi106.stdout:Package ceph-base-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 is already installed. 2024-04-04T20:57:57.306 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:57:57.307 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-04T20:57:57.307 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:57:57.367 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr-debuginfo 2024-04-04T20:57:57.696 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:57:57.697 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:57:57.929 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:08:02 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:57:58.078 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:57:58.079 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:57:58.079 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:57:58.079 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:57:58.079 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:57:58.079 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 20 M 2024-04-04T20:57:58.079 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:57:58.079 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:57:58.080 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:57:58.080 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:57:58.080 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:57:58.080 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 20 M 2024-04-04T20:57:58.080 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 104 M 2024-04-04T20:57:58.080 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:57:59.551 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-mgr-debuginfo-19.0.0-1864.g66deef15.el9.x8 14 MB/s | 20 MB 00:01 2024-04-04T20:57:59.552 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:57:59.553 INFO:teuthology.orchestra.run.smithi106.stdout:Total 14 MB/s | 20 MB 00:01 2024-04-04T20:57:59.553 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:57:59.557 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:57:59.557 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:57:59.709 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:57:59.709 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:58:01.971 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:58:02.076 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:58:03.016 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:58:03.270 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:58:03.270 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:58:03.270 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:58:03.270 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:58:03.270 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:58:03.270 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:58:03.436 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mds-debuginfo 2024-04-04T20:58:03.999 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:08:08 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:58:04.150 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:58:04.151 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:58:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:58:04.151 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:58:04.151 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:58:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 49 M 2024-04-04T20:58:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:58:04.151 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:58:04.151 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:58:04.151 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:58:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:58:04.151 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 49 M 2024-04-04T20:58:04.151 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 177 M 2024-04-04T20:58:04.152 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:58:06.796 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-mds-debuginfo-19.0.0-1864.g66deef15.el9.x8 18 MB/s | 49 MB 00:02 2024-04-04T20:58:06.796 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:58:06.797 INFO:teuthology.orchestra.run.smithi106.stdout:Total 18 MB/s | 49 MB 00:02 2024-04-04T20:58:06.797 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:58:06.801 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:58:06.801 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:58:07.123 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:58:07.124 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:58:07.961 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:58:08.067 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:58:11.195 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:58:11.460 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:58:11.461 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:58:11.461 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:58:11.461 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:58:11.461 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:58:11.461 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:58:11.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-osd-debuginfo 2024-04-04T20:58:11.972 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:58:12.061 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:58:12.241 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:08:07 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:58:12.392 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:58:12.393 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:58:12.393 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:58:12.393 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:58:12.393 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:58:12.394 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 288 M 2024-04-04T20:58:12.394 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:58:12.394 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:58:12.394 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:58:12.394 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:58:12.394 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:58:12.394 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 288 M 2024-04-04T20:58:12.394 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.2 G 2024-04-04T20:58:12.394 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:58:13.405 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:58:13.657 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:58:13.658 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:58:13.658 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:58:13.658 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:58:13.658 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:58:13.658 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:58:13.826 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mon-debuginfo 2024-04-04T20:58:14.395 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:08:19 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:58:14.549 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:58:14.550 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:58:14.550 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:58:14.550 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:58:14.550 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:58:14.550 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 99 M 2024-04-04T20:58:14.550 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:58:14.550 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:58:14.550 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:58:14.550 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:58:14.550 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:58:14.550 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 99 M 2024-04-04T20:58:14.551 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 450 M 2024-04-04T20:58:14.551 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:58:27.138 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-mon-debuginfo-19.0.0-1864.g66deef15.el9.x8 7.8 MB/s | 99 MB 00:12 2024-04-04T20:58:27.140 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:58:27.140 INFO:teuthology.orchestra.run.smithi106.stdout:Total 7.8 MB/s | 99 MB 00:12 2024-04-04T20:58:27.140 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:58:27.145 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:58:27.145 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:58:27.759 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:58:27.759 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:58:28.568 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-osd-debuginfo-19.0.0-1864.g66deef15.el9.x8 18 MB/s | 288 MB 00:16 2024-04-04T20:58:28.569 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:58:28.570 INFO:teuthology.orchestra.run.smithi203.stdout:Total 18 MB/s | 288 MB 00:16 2024-04-04T20:58:28.570 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:58:28.577 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:58:28.577 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:58:30.295 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:58:30.295 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:58:38.076 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:58:38.178 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:58:41.047 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:58:41.339 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:58:41.339 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:58:41.339 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:58:41.339 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:58:41.339 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:58:41.339 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:58:41.540 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-osd-debuginfo 2024-04-04T20:58:42.105 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:08:47 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:58:42.256 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:58:42.257 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:58:42.257 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:58:42.257 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:58:42.257 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:58:42.257 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 288 M 2024-04-04T20:58:42.257 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:58:42.257 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:58:42.257 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:58:42.257 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:58:42.258 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:58:42.258 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 288 M 2024-04-04T20:58:42.258 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 1.2 G 2024-04-04T20:58:42.258 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:58:54.864 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-osd-debuginfo-19.0.0-1864.g66deef15.el9.x8 23 MB/s | 288 MB 00:12 2024-04-04T20:58:54.865 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:58:54.865 INFO:teuthology.orchestra.run.smithi106.stdout:Total 23 MB/s | 288 MB 00:12 2024-04-04T20:58:54.866 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:58:54.870 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:58:54.870 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:58:56.594 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:58:56.594 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:58:59.828 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:59:00.115 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:59:08.107 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:59:08.392 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:59:08.392 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:08.392 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:59:08.392 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:08.392 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:08.392 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:59:08.630 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-fuse-debuginfo 2024-04-04T20:59:09.205 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:09:04 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:59:09.356 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:59:09.357 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:09.357 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:59:09.357 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:09.357 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:59:09.357 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 6.0 M 2024-04-04T20:59:09.358 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:09.358 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:59:09.358 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:09.358 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:59:09.358 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:09.358 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 6.0 M 2024-04-04T20:59:09.358 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 28 M 2024-04-04T20:59:09.358 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:59:09.876 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-fuse-debuginfo-19.0.0-1864.g66deef15.el9.x 12 MB/s | 6.0 MB 00:00 2024-04-04T20:59:09.877 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:09.877 INFO:teuthology.orchestra.run.smithi203.stdout:Total 11 MB/s | 6.0 MB 00:00 2024-04-04T20:59:09.878 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:59:09.881 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:59:09.881 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:59:09.950 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:59:09.951 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:59:10.685 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:59:10.784 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:59:11.360 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:59:11.626 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:59:11.627 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:11.627 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:59:11.627 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:11.627 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:11.627 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:59:11.828 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados-devel-debuginfo 2024-04-04T20:59:12.393 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:09:07 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:59:12.544 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:59:12.545 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:12.545 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:59:12.545 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:12.546 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:59:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 109 k 2024-04-04T20:59:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:12.546 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:59:12.546 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:12.546 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:59:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:12.546 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 109 k 2024-04-04T20:59:12.546 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 440 k 2024-04-04T20:59:12.546 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:59:12.863 INFO:teuthology.orchestra.run.smithi203.stdout:librados-devel-debuginfo-19.0.0-1864.g66deef15. 343 kB/s | 109 kB 00:00 2024-04-04T20:59:12.864 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:12.864 INFO:teuthology.orchestra.run.smithi203.stdout:Total 342 kB/s | 109 kB 00:00 2024-04-04T20:59:12.864 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:59:12.867 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:59:12.867 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:59:12.903 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:59:12.904 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:59:13.020 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:59:13.121 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1864.g66deef15.el9 1/1 2024-04-04T20:59:13.511 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1864.g66deef15.el9 1/1 2024-04-04T20:59:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1864.g66deef15.el9 1/1 2024-04-04T20:59:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:13.783 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:59:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:13.784 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:59:13.980 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs2-debuginfo 2024-04-04T20:59:14.549 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:09:09 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:59:14.699 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:59:14.700 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:14.700 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:59:14.700 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:14.700 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:59:14.700 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 5.4 M 2024-04-04T20:59:14.700 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:14.700 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:59:14.700 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:14.700 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:59:14.700 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:14.701 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 5.4 M 2024-04-04T20:59:14.701 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 25 M 2024-04-04T20:59:14.701 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:59:15.367 INFO:teuthology.orchestra.run.smithi203.stdout:libcephfs2-debuginfo-19.0.0-1864.g66deef15.el9. 8.1 MB/s | 5.4 MB 00:00 2024-04-04T20:59:15.367 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:15.368 INFO:teuthology.orchestra.run.smithi203.stdout:Total 8.1 MB/s | 5.4 MB 00:00 2024-04-04T20:59:15.368 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:59:15.372 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:59:15.372 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:59:15.437 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:59:15.437 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:59:16.108 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:59:16.207 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86 1/1 2024-04-04T20:59:16.770 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86 1/1 2024-04-04T20:59:17.049 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86 1/1 2024-04-04T20:59:17.050 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:17.050 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:59:17.050 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:17.050 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:17.050 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:59:17.251 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados2-debuginfo 2024-04-04T20:59:17.815 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:09:12 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:59:17.966 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:59:17.967 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:17.967 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:59:17.967 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:17.967 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:59:17.967 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 45 M 2024-04-04T20:59:17.967 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:17.967 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:59:17.967 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:17.967 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:59:17.967 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:17.967 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 45 M 2024-04-04T20:59:17.967 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 255 M 2024-04-04T20:59:17.968 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:59:20.303 INFO:teuthology.orchestra.run.smithi203.stdout:librados2-debuginfo-19.0.0-1864.g66deef15.el9.x 19 MB/s | 45 MB 00:02 2024-04-04T20:59:20.304 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:20.304 INFO:teuthology.orchestra.run.smithi203.stdout:Total 19 MB/s | 45 MB 00:02 2024-04-04T20:59:20.305 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:59:20.309 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:59:20.309 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:59:20.615 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:59:20.615 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:59:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:59:25.852 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:59:26.247 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:59:26.354 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:59:27.930 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:59:28.188 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:59:28.189 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:28.189 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:59:28.189 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:28.189 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:28.189 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:59:28.403 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librbd1-debuginfo 2024-04-04T20:59:28.973 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:09:23 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:59:29.123 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:59:29.123 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:29.124 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:59:29.124 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:29.124 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:59:29.124 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 52 M 2024-04-04T20:59:29.124 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:29.124 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:59:29.124 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:29.124 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:59:29.124 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:29.124 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 52 M 2024-04-04T20:59:29.124 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 308 M 2024-04-04T20:59:29.124 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:59:33.323 INFO:teuthology.orchestra.run.smithi203.stdout:librbd1-debuginfo-19.0.0-1864.g66deef15.el9.x86 12 MB/s | 52 MB 00:04 2024-04-04T20:59:33.324 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:33.325 INFO:teuthology.orchestra.run.smithi203.stdout:Total 12 MB/s | 52 MB 00:04 2024-04-04T20:59:33.325 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:59:33.329 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:59:33.329 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:59:33.677 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:59:33.677 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:59:33.833 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:59:34.108 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:59:34.109 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:34.109 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:59:34.109 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:34.109 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:34.109 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:59:34.304 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-fuse-debuginfo 2024-04-04T20:59:34.873 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:09:39 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:59:35.025 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:59:35.025 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:35.025 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:59:35.025 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:35.025 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:59:35.025 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 6.0 M 2024-04-04T20:59:35.025 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:35.025 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:59:35.025 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:35.025 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:59:35.026 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:35.026 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 6.0 M 2024-04-04T20:59:35.026 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 28 M 2024-04-04T20:59:35.026 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:59:35.690 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-fuse-debuginfo-19.0.0-1864.g66deef15.el9.x 9.0 MB/s | 6.0 MB 00:00 2024-04-04T20:59:35.690 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:35.690 INFO:teuthology.orchestra.run.smithi106.stdout:Total 9.0 MB/s | 6.0 MB 00:00 2024-04-04T20:59:35.691 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:59:35.694 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:59:35.694 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:59:35.766 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:59:35.766 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:59:36.498 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:59:36.588 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:59:37.150 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:59:37.408 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:59:37.408 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:37.408 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:59:37.408 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:37.408 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:37.408 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:59:37.606 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librados-devel-debuginfo 2024-04-04T20:59:38.174 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:09:43 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:59:38.325 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:59:38.326 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:38.326 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:59:38.326 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:38.327 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:59:38.327 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 109 k 2024-04-04T20:59:38.327 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:38.327 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:59:38.327 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:38.327 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:59:38.327 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:38.327 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 109 k 2024-04-04T20:59:38.327 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 440 k 2024-04-04T20:59:38.328 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:59:38.643 INFO:teuthology.orchestra.run.smithi106.stdout:librados-devel-debuginfo-19.0.0-1864.g66deef15. 345 kB/s | 109 kB 00:00 2024-04-04T20:59:38.643 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:38.644 INFO:teuthology.orchestra.run.smithi106.stdout:Total 343 kB/s | 109 kB 00:00 2024-04-04T20:59:38.644 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:59:38.647 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:59:38.647 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:59:38.682 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:59:38.683 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:59:38.797 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:59:38.908 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1864.g66deef15.el9 1/1 2024-04-04T20:59:39.274 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1864.g66deef15.el9 1/1 2024-04-04T20:59:39.531 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1864.g66deef15.el9 1/1 2024-04-04T20:59:39.531 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:39.531 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:59:39.531 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:39.531 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:39.531 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:59:39.681 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install libcephfs2-debuginfo 2024-04-04T20:59:39.701 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:59:39.796 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librbd1-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:59:40.244 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:09:45 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:59:40.395 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:59:40.395 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:40.395 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:59:40.395 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:40.395 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:59:40.395 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 5.4 M 2024-04-04T20:59:40.395 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:40.395 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:59:40.396 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:40.396 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:59:40.396 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:40.396 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 5.4 M 2024-04-04T20:59:40.396 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 25 M 2024-04-04T20:59:40.396 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:59:40.879 INFO:teuthology.orchestra.run.smithi106.stdout:libcephfs2-debuginfo-19.0.0-1864.g66deef15.el9. 11 MB/s | 5.4 MB 00:00 2024-04-04T20:59:40.879 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:40.880 INFO:teuthology.orchestra.run.smithi106.stdout:Total 11 MB/s | 5.4 MB 00:00 2024-04-04T20:59:40.880 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:59:40.883 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:59:40.883 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:59:40.948 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:59:40.949 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:59:41.619 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:59:41.736 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86 1/1 2024-04-04T20:59:42.086 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:59:42.242 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86 1/1 2024-04-04T20:59:42.372 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T20:59:42.372 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:42.372 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:59:42.372 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:42.372 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:42.372 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:59:42.497 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86 1/1 2024-04-04T20:59:42.498 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:42.498 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:59:42.498 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:42.498 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:42.498 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:59:42.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-cephfs-debuginfo 2024-04-04T20:59:42.687 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librados2-debuginfo 2024-04-04T20:59:43.166 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:09:38 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:59:43.249 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:09:48 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:59:43.317 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:59:43.318 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:43.318 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:59:43.318 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:43.319 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:59:43.319 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 615 k 2024-04-04T20:59:43.319 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:43.319 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:59:43.319 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:43.319 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:59:43.319 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:43.319 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 615 k 2024-04-04T20:59:43.319 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.1 M 2024-04-04T20:59:43.319 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:59:43.400 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:59:43.401 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:43.401 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:59:43.401 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:43.401 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:59:43.401 INFO:teuthology.orchestra.run.smithi106.stdout: librados2-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 45 M 2024-04-04T20:59:43.401 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:43.401 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:59:43.401 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:43.401 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:59:43.401 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:43.401 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 45 M 2024-04-04T20:59:43.401 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 255 M 2024-04-04T20:59:43.402 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:59:43.743 INFO:teuthology.orchestra.run.smithi203.stdout:python3-cephfs-debuginfo-19.0.0-1864.g66deef15. 1.4 MB/s | 615 kB 00:00 2024-04-04T20:59:43.743 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:43.743 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.4 MB/s | 615 kB 00:00 2024-04-04T20:59:43.743 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:59:43.747 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:59:43.747 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:59:43.785 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:59:43.785 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:59:43.957 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:59:44.061 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1864.g66deef15.el9 1/1 2024-04-04T20:59:44.465 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1864.g66deef15.el9 1/1 2024-04-04T20:59:44.747 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1864.g66deef15.el9 1/1 2024-04-04T20:59:44.747 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:44.747 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:59:44.747 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:44.747 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:44.747 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:59:44.944 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rados-debuginfo 2024-04-04T20:59:45.510 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:09:40 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:59:45.665 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:59:45.666 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:45.666 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:59:45.666 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:45.666 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:59:45.666 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 770 k 2024-04-04T20:59:45.666 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:45.666 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:59:45.666 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:45.666 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:59:45.666 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:45.666 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 770 k 2024-04-04T20:59:45.666 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.7 M 2024-04-04T20:59:45.666 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:59:45.748 INFO:teuthology.orchestra.run.smithi106.stdout:librados2-debuginfo-19.0.0-1864.g66deef15.el9.x 19 MB/s | 45 MB 00:02 2024-04-04T20:59:45.749 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:45.750 INFO:teuthology.orchestra.run.smithi106.stdout:Total 19 MB/s | 45 MB 00:02 2024-04-04T20:59:45.750 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:59:45.754 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:59:45.754 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:59:45.939 INFO:teuthology.orchestra.run.smithi203.stdout:python3-rados-debuginfo-19.0.0-1864.g66deef15.e 2.8 MB/s | 770 kB 00:00 2024-04-04T20:59:45.939 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:45.939 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.8 MB/s | 770 kB 00:00 2024-04-04T20:59:45.939 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:59:45.942 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:59:45.943 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:59:45.981 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:59:45.981 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:59:46.058 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:59:46.059 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T20:59:46.159 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:59:46.271 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1864.g66deef15.el9. 1/1 2024-04-04T20:59:46.700 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1864.g66deef15.el9. 1/1 2024-04-04T20:59:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1864.g66deef15.el9. 1/1 2024-04-04T20:59:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:46.978 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:59:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:46.978 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:46.978 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:59:47.207 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rbd-debuginfo 2024-04-04T20:59:47.774 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:09:42 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:59:47.928 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:59:47.929 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:47.929 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:59:47.929 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:47.929 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:59:47.929 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 1.1 M 2024-04-04T20:59:47.929 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:47.929 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:59:47.929 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:47.929 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:59:47.929 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:47.930 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 1.1 M 2024-04-04T20:59:47.930 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 4.3 M 2024-04-04T20:59:47.930 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:59:48.517 INFO:teuthology.orchestra.run.smithi203.stdout:python3-rbd-debuginfo-19.0.0-1864.g66deef15.el9 1.9 MB/s | 1.1 MB 00:00 2024-04-04T20:59:48.517 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:48.517 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.9 MB/s | 1.1 MB 00:00 2024-04-04T20:59:48.518 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:59:48.521 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:59:48.521 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:59:48.563 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:59:48.563 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:59:48.784 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:59:48.893 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T20:59:49.285 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T20:59:49.552 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T20:59:49.552 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:49.552 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:59:49.552 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:49.552 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:49.552 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:59:49.749 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rgw-debuginfo 2024-04-04T20:59:50.320 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:09:45 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:59:50.471 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:59:50.472 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:50.472 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:59:50.472 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:50.472 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:59:50.472 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 337 k 2024-04-04T20:59:50.472 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:50.472 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:59:50.472 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:50.472 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:59:50.472 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:50.473 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 337 k 2024-04-04T20:59:50.473 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.0 M 2024-04-04T20:59:50.473 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:59:50.721 INFO:teuthology.orchestra.run.smithi203.stdout:python3-rgw-debuginfo-19.0.0-1864.g66deef15.el9 1.3 MB/s | 337 kB 00:00 2024-04-04T20:59:50.722 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:50.722 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.3 MB/s | 337 kB 00:00 2024-04-04T20:59:50.722 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:59:50.725 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:59:50.725 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:59:50.762 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:59:50.762 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:59:50.902 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:59:51.012 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T20:59:51.162 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T20:59:51.255 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librados2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:59:51.430 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T20:59:51.702 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T20:59:51.703 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:51.703 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:59:51.703 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:51.703 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:51.703 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:59:51.897 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-fuse-debuginfo 2024-04-04T20:59:52.461 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:09:47 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:59:52.613 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:59:52.613 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:52.613 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:59:52.613 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:52.614 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:59:52.614 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 516 k 2024-04-04T20:59:52.614 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:52.614 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:59:52.614 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:52.614 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:59:52.614 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:52.614 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 516 k 2024-04-04T20:59:52.614 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.6 M 2024-04-04T20:59:52.614 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:59:53.090 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:59:53.216 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-fuse-debuginfo-19.0.0-1864.g66deef15.el9.x8 859 kB/s | 516 kB 00:00 2024-04-04T20:59:53.217 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:53.217 INFO:teuthology.orchestra.run.smithi203.stdout:Total 857 kB/s | 516 kB 00:00 2024-04-04T20:59:53.217 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T20:59:53.220 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T20:59:53.220 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T20:59:53.260 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T20:59:53.260 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T20:59:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_ 1/1 2024-04-04T20:59:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:53.339 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T20:59:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: librados2-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:53.339 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T20:59:53.441 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T20:59:53.512 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librbd1-debuginfo 2024-04-04T20:59:53.549 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:59:53.934 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:59:54.094 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:09:59 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T20:59:54.201 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T20:59:54.201 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:54.202 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T20:59:54.202 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T20:59:54.202 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:54.202 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T20:59:54.256 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T20:59:54.257 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:54.257 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T20:59:54.257 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:54.257 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T20:59:54.257 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 52 M 2024-04-04T20:59:54.257 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:54.257 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T20:59:54.257 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T20:59:54.257 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T20:59:54.257 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T20:59:54.258 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 52 M 2024-04-04T20:59:54.258 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 308 M 2024-04-04T20:59:54.258 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T20:59:54.414 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-mirror-debuginfo 2024-04-04T20:59:54.983 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:09:49 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T20:59:55.135 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T20:59:55.135 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:55.136 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T20:59:55.136 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:55.136 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T20:59:55.136 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 52 M 2024-04-04T20:59:55.136 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:55.136 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T20:59:55.136 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T20:59:55.136 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T20:59:55.136 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T20:59:55.136 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 52 M 2024-04-04T20:59:55.136 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 298 M 2024-04-04T20:59:55.136 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T20:59:57.418 INFO:teuthology.orchestra.run.smithi106.stdout:librbd1-debuginfo-19.0.0-1864.g66deef15.el9.x86 16 MB/s | 52 MB 00:03 2024-04-04T20:59:57.418 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T20:59:57.419 INFO:teuthology.orchestra.run.smithi106.stdout:Total 16 MB/s | 52 MB 00:03 2024-04-04T20:59:57.419 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T20:59:57.423 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T20:59:57.423 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T20:59:57.772 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T20:59:57.772 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T21:00:01.136 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-mirror-debuginfo-19.0.0-1864.g66deef15.el9. 8.6 MB/s | 52 MB 00:05 2024-04-04T21:00:01.137 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T21:00:01.137 INFO:teuthology.orchestra.run.smithi203.stdout:Total 8.6 MB/s | 52 MB 00:06 2024-04-04T21:00:01.138 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T21:00:01.142 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T21:00:01.142 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T21:00:01.479 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T21:00:01.480 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T21:00:03.804 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T21:00:03.899 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librbd1-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T21:00:06.127 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T21:00:06.386 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T21:00:06.386 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:06.386 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T21:00:06.386 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T21:00:06.386 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:06.386 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T21:00:06.566 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-cephfs-debuginfo 2024-04-04T21:00:07.139 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:10:12 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T21:00:07.291 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T21:00:07.292 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:07.292 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T21:00:07.292 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:07.292 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T21:00:07.292 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 615 k 2024-04-04T21:00:07.293 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:07.293 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T21:00:07.293 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:07.293 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T21:00:07.293 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:07.293 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 615 k 2024-04-04T21:00:07.293 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 2.1 M 2024-04-04T21:00:07.293 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T21:00:07.396 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T21:00:07.492 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1864.g66deef15.el9.x86 1/1 2024-04-04T21:00:07.562 INFO:teuthology.orchestra.run.smithi106.stdout:python3-cephfs-debuginfo-19.0.0-1864.g66deef15. 2.2 MB/s | 615 kB 00:00 2024-04-04T21:00:07.563 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T21:00:07.564 INFO:teuthology.orchestra.run.smithi106.stdout:Total 2.2 MB/s | 615 kB 00:00 2024-04-04T21:00:07.564 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T21:00:07.567 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T21:00:07.567 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T21:00:07.605 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T21:00:07.606 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T21:00:07.770 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T21:00:07.870 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1864.g66deef15.el9 1/1 2024-04-04T21:00:08.238 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1864.g66deef15.el9 1/1 2024-04-04T21:00:08.494 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1864.g66deef15.el9 1/1 2024-04-04T21:00:08.494 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:08.494 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T21:00:08.494 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T21:00:08.495 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:08.495 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T21:00:08.689 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-rados-debuginfo 2024-04-04T21:00:09.255 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:10:14 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T21:00:09.404 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T21:00:09.405 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:09.405 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T21:00:09.405 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:09.405 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T21:00:09.405 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 770 k 2024-04-04T21:00:09.405 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:09.405 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T21:00:09.405 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:09.406 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T21:00:09.406 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:09.406 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 770 k 2024-04-04T21:00:09.406 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 2.7 M 2024-04-04T21:00:09.406 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T21:00:09.664 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1864.g66deef15.el9.x86 1/1 2024-04-04T21:00:09.690 INFO:teuthology.orchestra.run.smithi106.stdout:python3-rados-debuginfo-19.0.0-1864.g66deef15.e 2.6 MB/s | 770 kB 00:00 2024-04-04T21:00:09.691 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T21:00:09.691 INFO:teuthology.orchestra.run.smithi106.stdout:Total 2.6 MB/s | 770 kB 00:00 2024-04-04T21:00:09.691 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T21:00:09.694 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T21:00:09.694 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T21:00:09.732 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T21:00:09.733 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T21:00:09.914 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T21:00:09.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1864.g66deef15.el9.x86 1/1 2024-04-04T21:00:09.931 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:09.931 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T21:00:09.932 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T21:00:09.932 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:09.932 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T21:00:10.013 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1864.g66deef15.el9. 1/1 2024-04-04T21:00:10.131 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-nbd-debuginfo 2024-04-04T21:00:10.364 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1864.g66deef15.el9. 1/1 2024-04-04T21:00:10.643 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1864.g66deef15.el9. 1/1 2024-04-04T21:00:10.643 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:10.643 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T21:00:10.643 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T21:00:10.643 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:10.643 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T21:00:10.698 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:10:05 ago on Thu 04 Apr 2024 08:50:05 PM UTC. 2024-04-04T21:00:10.851 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-04T21:00:10.852 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T21:00:10.852 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-04T21:00:10.853 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T21:00:10.853 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-04T21:00:10.853 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 1.3 M 2024-04-04T21:00:10.853 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:10.853 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-04T21:00:10.853 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-04T21:00:10.853 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-04T21:00:10.853 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:10.853 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 1.3 M 2024-04-04T21:00:10.853 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 5.7 M 2024-04-04T21:00:10.854 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-04T21:00:10.864 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-rbd-debuginfo 2024-04-04T21:00:11.188 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-nbd-debuginfo-19.0.0-1864.g66deef15.el9.x86 3.8 MB/s | 1.3 MB 00:00 2024-04-04T21:00:11.189 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-04T21:00:11.189 INFO:teuthology.orchestra.run.smithi203.stdout:Total 3.8 MB/s | 1.3 MB 00:00 2024-04-04T21:00:11.189 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-04T21:00:11.193 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-04T21:00:11.193 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-04T21:00:11.234 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-04T21:00:11.234 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-04T21:00:11.434 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:10:16 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T21:00:11.479 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-04T21:00:11.580 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T21:00:11.585 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T21:00:11.585 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:11.586 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T21:00:11.586 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:11.586 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T21:00:11.586 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 1.1 M 2024-04-04T21:00:11.586 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:11.586 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T21:00:11.586 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:11.586 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T21:00:11.586 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:11.590 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 1.1 M 2024-04-04T21:00:11.590 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 4.3 M 2024-04-04T21:00:11.590 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T21:00:11.885 INFO:teuthology.orchestra.run.smithi106.stdout:python3-rbd-debuginfo-19.0.0-1864.g66deef15.el9 3.8 MB/s | 1.1 MB 00:00 2024-04-04T21:00:11.886 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T21:00:11.886 INFO:teuthology.orchestra.run.smithi106.stdout:Total 3.8 MB/s | 1.1 MB 00:00 2024-04-04T21:00:11.887 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T21:00:11.890 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T21:00:11.890 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T21:00:11.930 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T21:00:11.931 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T21:00:12.024 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T21:00:12.145 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T21:00:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T21:00:12.307 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T21:00:12.308 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:12.308 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-04T21:00:12.308 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T21:00:12.308 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:12.308 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-04T21:00:12.526 DEBUG:teuthology.parallel:result is None 2024-04-04T21:00:12.645 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T21:00:12.911 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T21:00:12.911 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:12.911 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T21:00:12.911 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T21:00:12.912 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:12.912 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T21:00:13.106 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-rgw-debuginfo 2024-04-04T21:00:13.671 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:10:18 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T21:00:13.820 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T21:00:13.821 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:13.821 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T21:00:13.821 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:13.822 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T21:00:13.822 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 337 k 2024-04-04T21:00:13.822 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:13.822 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T21:00:13.822 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:13.822 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T21:00:13.822 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:13.822 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 337 k 2024-04-04T21:00:13.822 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 1.0 M 2024-04-04T21:00:13.822 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T21:00:14.213 INFO:teuthology.orchestra.run.smithi106.stdout:python3-rgw-debuginfo-19.0.0-1864.g66deef15.el9 863 kB/s | 337 kB 00:00 2024-04-04T21:00:14.214 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T21:00:14.214 INFO:teuthology.orchestra.run.smithi106.stdout:Total 860 kB/s | 337 kB 00:00 2024-04-04T21:00:14.214 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T21:00:14.218 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T21:00:14.218 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T21:00:14.253 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T21:00:14.254 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T21:00:14.392 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T21:00:14.486 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T21:00:14.857 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T21:00:15.117 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x8 1/1 2024-04-04T21:00:15.118 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:15.118 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T21:00:15.118 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T21:00:15.118 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:15.118 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T21:00:15.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-fuse-debuginfo 2024-04-04T21:00:15.902 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:10:20 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T21:00:16.053 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T21:00:16.054 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:16.054 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T21:00:16.054 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:16.054 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T21:00:16.055 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 516 k 2024-04-04T21:00:16.055 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:16.055 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T21:00:16.055 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:16.055 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T21:00:16.055 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:16.056 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 516 k 2024-04-04T21:00:16.056 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 2.6 M 2024-04-04T21:00:16.056 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T21:00:16.341 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-fuse-debuginfo-19.0.0-1864.g66deef15.el9.x8 1.8 MB/s | 516 kB 00:00 2024-04-04T21:00:16.342 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T21:00:16.342 INFO:teuthology.orchestra.run.smithi106.stdout:Total 1.7 MB/s | 516 kB 00:00 2024-04-04T21:00:16.342 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T21:00:16.346 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T21:00:16.346 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T21:00:16.383 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T21:00:16.383 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T21:00:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T21:00:16.657 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T21:00:17.079 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T21:00:17.335 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_6 1/1 2024-04-04T21:00:17.335 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:17.335 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T21:00:17.335 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T21:00:17.336 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:17.336 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T21:00:17.509 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-mirror-debuginfo 2024-04-04T21:00:18.075 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:10:23 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T21:00:18.227 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T21:00:18.228 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:18.228 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T21:00:18.228 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:18.228 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T21:00:18.228 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 52 M 2024-04-04T21:00:18.229 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:18.229 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T21:00:18.229 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:18.229 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T21:00:18.229 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:18.229 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 52 M 2024-04-04T21:00:18.229 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 298 M 2024-04-04T21:00:18.229 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T21:00:20.670 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-mirror-debuginfo-19.0.0-1864.g66deef15.el9. 21 MB/s | 52 MB 00:02 2024-04-04T21:00:20.670 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T21:00:20.671 INFO:teuthology.orchestra.run.smithi106.stdout:Total 21 MB/s | 52 MB 00:02 2024-04-04T21:00:20.671 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T21:00:20.675 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T21:00:20.675 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T21:00:21.015 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T21:00:21.015 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T21:00:26.931 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T21:00:27.019 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1864.g66deef15.el9.x86 1/1 2024-04-04T21:00:29.194 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1864.g66deef15.el9.x86 1/1 2024-04-04T21:00:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1864.g66deef15.el9.x86 1/1 2024-04-04T21:00:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:29.449 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T21:00:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T21:00:29.449 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:29.449 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T21:00:29.627 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-nbd-debuginfo 2024-04-04T21:00:30.194 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:10:35 ago on Thu 04 Apr 2024 08:49:55 PM UTC. 2024-04-04T21:00:30.345 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-04T21:00:30.346 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:30.346 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-04-04T21:00:30.346 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:30.346 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-04T21:00:30.346 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1864.g66deef15.el9 ceph 1.3 M 2024-04-04T21:00:30.346 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:30.346 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-04T21:00:30.346 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-04T21:00:30.347 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-04T21:00:30.347 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:30.347 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 1.3 M 2024-04-04T21:00:30.347 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 5.7 M 2024-04-04T21:00:30.347 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-04T21:00:30.822 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-nbd-debuginfo-19.0.0-1864.g66deef15.el9.x86 2.7 MB/s | 1.3 MB 00:00 2024-04-04T21:00:30.822 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-04T21:00:30.823 INFO:teuthology.orchestra.run.smithi106.stdout:Total 2.7 MB/s | 1.3 MB 00:00 2024-04-04T21:00:30.823 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-04T21:00:30.826 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-04T21:00:30.826 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-04T21:00:30.867 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-04T21:00:30.868 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-04T21:00:31.108 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-04T21:00:31.222 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T21:00:31.606 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T21:00:31.874 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 1/1 2024-04-04T21:00:31.875 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:31.875 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-04T21:00:31.875 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd-debuginfo-2:19.0.0-1864.g66deef15.el9.x86_64 2024-04-04T21:00:31.875 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:31.875 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-04T21:00:32.054 DEBUG:teuthology.parallel:result is None 2024-04-04T21:00:32.055 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=66deef1598934c3b1ef2f044e380a7d00244cead 2024-04-04T21:00:32.276 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-04T21:00:32.297 INFO:teuthology.orchestra.run.smithi106.stdout:19.0.0-1864.g66deef15.el9 2024-04-04T21:00:32.298 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1864.g66deef15.el9 2024-04-04T21:00:32.298 INFO:teuthology.task.install:The correct ceph version 19.0.0-1864.g66deef15 is installed. 2024-04-04T21:00:32.300 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=66deef1598934c3b1ef2f044e380a7d00244cead 2024-04-04T21:00:32.444 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-04T21:00:32.469 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-1864.g66deef15.el9 2024-04-04T21:00:32.470 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1864.g66deef15.el9 2024-04-04T21:00:32.470 INFO:teuthology.task.install:The correct ceph version 19.0.0-1864.g66deef15 is installed. 2024-04-04T21:00:32.471 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-04T21:00:32.472 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:00:32.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-04T21:00:32.499 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:00:32.499 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-04T21:00:32.541 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-04T21:00:32.542 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:00:32.542 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/daemon-helper 2024-04-04T21:00:32.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-04T21:00:32.635 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:00:32.635 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-04-04T21:00:32.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-04T21:00:32.736 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-04T21:00:32.736 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:00:32.736 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-04T21:00:32.763 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-04T21:00:32.830 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:00:32.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-04T21:00:32.861 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-04T21:00:32.932 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-04T21:00:32.932 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:00:32.933 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/stdin-killer 2024-04-04T21:00:32.959 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-04T21:00:33.026 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:00:33.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-04-04T21:00:33.055 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-04T21:00:33.126 INFO:teuthology.run_tasks:Running task ceph... 2024-04-04T21:00:33.220 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-04T21:00:33.220 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 777 /var/log/ceph 2024-04-04T21:00:33.223 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /var/log/ceph 2024-04-04T21:00:33.253 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-04T21:00:33.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-04T21:00:33.290 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-04T21:00:33.321 INFO:tasks.ceph:Creating extra log directories... 2024-04-04T21:00:33.322 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-04T21:00:33.356 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-04T21:00:33.400 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-04T21:00:33.401 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-04T21:00:33.401 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121', 'branch': 'wip-ceph_test_rados-partial-reads', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7640121', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-ceph_test_rados-partial-reads'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '66deef1598934c3b1ef2f044e380a7d00244cead', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '66deef1598934c3b1ef2f044e380a7d00244cead'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-ceph_test_rados-partial-reads', 'sha1': '66deef1598934c3b1ef2f044e380a7d00244cead'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 3679, 'sha1': '66deef1598934c3b1ef2f044e380a7d00244cead', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-ceph_test_rados-partial-reads', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '66deef1598934c3b1ef2f044e380a7d00244cead', 'targets': {'smithi106.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO5pKSz3FNac+x2k6GP1zszcrUyY03hsljHicGucMoj4mXJbN0r0LR7iGE+oQekIZlxoMd2ak7lpFdpiLOTaUQQ=', 'smithi203.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBqwBEST0LdPtiFBL4rbRllWhJoAZHrBC6yTmDPvJRG/iuIvA4abrIenfHOl7SxciGjcSoBIQe6fuYuK6YSMIqA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'd7d6e60867cadf19fbfd7655a551ae7d58060757', 'timestamp': '2024-04-04_14:50:58', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-04T21:00:33.401 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-04T21:00:33.423 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-04T21:00:33.447 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m0777 -- /var/run/ceph 2024-04-04T21:00:33.479 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/run/ceph 2024-04-04T21:00:33.517 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:00:33.517 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2024-04-04T21:00:33.558 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-04T21:00:33.558 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_1 2024-04-04T21:00:33.613 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-04T21:00:33.613 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-04T21:00:33.613 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-04T21:00:33.613 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-04T21:00:33.613 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-04-04T21:00:33.614 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-04-04 20:51:46.187156163 +0000 2024-04-04T21:00:33.614 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-04-04 20:49:07.787916201 +0000 2024-04-04T21:00:33.614 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-04-04 20:49:07.787916201 +0000 2024-04-04T21:00:33.614 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-04-04 20:49:07.787916201 +0000 2024-04-04T21:00:33.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-04T21:00:33.679 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-04-04T21:00:33.679 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-04-04T21:00:33.679 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000199773 s, 2.6 MB/s 2024-04-04T21:00:33.680 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-04T21:00:33.736 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_2 2024-04-04T21:00:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-04T21:00:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-04T21:00:33.791 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-04T21:00:33.791 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-04T21:00:33.792 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-04-04T21:00:33.792 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-04-04 20:51:46.187156163 +0000 2024-04-04T21:00:33.792 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-04-04 20:49:08.135907436 +0000 2024-04-04T21:00:33.792 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-04-04 20:49:08.135907436 +0000 2024-04-04T21:00:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-04-04 20:49:08.135907436 +0000 2024-04-04T21:00:33.792 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-04T21:00:33.858 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-04-04T21:00:33.858 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-04-04T21:00:33.858 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000292967 s, 1.7 MB/s 2024-04-04T21:00:33.859 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-04T21:00:33.917 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_3 2024-04-04T21:00:33.972 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-04T21:00:33.973 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-04T21:00:33.973 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-04T21:00:33.973 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-04T21:00:33.973 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-04-04T21:00:33.973 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-04-04 20:51:46.188156138 +0000 2024-04-04T21:00:33.973 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-04-04 20:49:08.471898973 +0000 2024-04-04T21:00:33.973 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-04-04 20:49:08.471898973 +0000 2024-04-04T21:00:33.973 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-04-04 20:49:08.471898973 +0000 2024-04-04T21:00:33.974 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-04T21:00:34.038 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-04-04T21:00:34.038 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-04-04T21:00:34.038 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000204935 s, 2.5 MB/s 2024-04-04T21:00:34.039 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-04T21:00:34.095 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_4 2024-04-04T21:00:34.150 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-04T21:00:34.151 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-04T21:00:34.151 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-04T21:00:34.151 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-04T21:00:34.151 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-04-04T21:00:34.151 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-04-04 20:51:46.188156138 +0000 2024-04-04T21:00:34.151 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-04-04 20:49:08.818890233 +0000 2024-04-04T21:00:34.151 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-04-04 20:49:08.818890233 +0000 2024-04-04T21:00:34.151 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-04-04 20:49:08.818890233 +0000 2024-04-04T21:00:34.151 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-04T21:00:34.215 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-04-04T21:00:34.215 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-04-04T21:00:34.216 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000172301 s, 3.0 MB/s 2024-04-04T21:00:34.217 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-04T21:00:34.273 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-04T21:00:34.273 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:00:34.273 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-04-04T21:00:34.291 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-04T21:00:34.292 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-04-04T21:00:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-04T21:00:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-04T21:00:34.350 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-04T21:00:34.350 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-04T21:00:34.350 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-04T21:00:34.350 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-04 20:51:52.922857697 +0000 2024-04-04T21:00:34.350 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-04 20:49:07.795579625 +0000 2024-04-04T21:00:34.351 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-04 20:49:07.795579625 +0000 2024-04-04T21:00:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-04 20:49:07.795579625 +0000 2024-04-04T21:00:34.351 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-04T21:00:34.419 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-04T21:00:34.419 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-04T21:00:34.419 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000221811 s, 2.3 MB/s 2024-04-04T21:00:34.421 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-04T21:00:34.479 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-04-04T21:00:34.536 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-04T21:00:34.536 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-04T21:00:34.537 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-04T21:00:34.537 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-04T21:00:34.537 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-04T21:00:34.537 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-04 20:51:52.923857672 +0000 2024-04-04T21:00:34.537 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-04 20:49:08.133571007 +0000 2024-04-04T21:00:34.537 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-04 20:49:08.133571007 +0000 2024-04-04T21:00:34.537 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-04 20:49:08.133571007 +0000 2024-04-04T21:00:34.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-04T21:00:34.604 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-04T21:00:34.604 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-04T21:00:34.604 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000233061 s, 2.2 MB/s 2024-04-04T21:00:34.606 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-04T21:00:34.663 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-04-04T21:00:34.719 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-04T21:00:34.719 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-04T21:00:34.719 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-04T21:00:34.719 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-04T21:00:34.719 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-04T21:00:34.719 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-04 20:51:52.923857672 +0000 2024-04-04T21:00:34.719 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-04 20:49:08.475562287 +0000 2024-04-04T21:00:34.719 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-04 20:49:08.475562287 +0000 2024-04-04T21:00:34.719 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-04 20:49:08.475562287 +0000 2024-04-04T21:00:34.720 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-04T21:00:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-04T21:00:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-04T21:00:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000195559 s, 2.6 MB/s 2024-04-04T21:00:34.788 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-04T21:00:34.844 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-04-04T21:00:34.899 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-04T21:00:34.900 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-04T21:00:34.900 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-04T21:00:34.900 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-04T21:00:34.900 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-04T21:00:34.900 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-04 20:51:52.923857672 +0000 2024-04-04T21:00:34.900 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-04 20:49:08.818553541 +0000 2024-04-04T21:00:34.900 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-04 20:49:08.818553541 +0000 2024-04-04T21:00:34.900 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-04 20:49:08.818553541 +0000 2024-04-04T21:00:34.900 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-04T21:00:34.966 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-04T21:00:34.966 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-04T21:00:34.966 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000196572 s, 2.6 MB/s 2024-04-04T21:00:34.967 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-04T21:00:35.024 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-04T21:00:35.024 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi106.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-04T21:00:35.024 INFO:tasks.ceph:Generating config... 2024-04-04T21:00:35.026 INFO:tasks.ceph:[client] debug monc = 20 2024-04-04T21:00:35.026 INFO:tasks.ceph:[client] debug ms = 1 2024-04-04T21:00:35.026 INFO:tasks.ceph:[client] debug objecter = 20 2024-04-04T21:00:35.026 INFO:tasks.ceph:[client] debug rados = 20 2024-04-04T21:00:35.026 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-04T21:00:35.026 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-04T21:00:35.026 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-04T21:00:35.026 INFO:tasks.ceph:[global] ms type = async 2024-04-04T21:00:35.026 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-04-04T21:00:35.026 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-04T21:00:35.026 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-04T21:00:35.026 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-04T21:00:35.027 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-04T21:00:35.027 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-04T21:00:35.027 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-04T21:00:35.027 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-04-04T21:00:35.027 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-04T21:00:35.027 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-04T21:00:35.027 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-04T21:00:35.027 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-04T21:00:35.027 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-04-04T21:00:35.027 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-04T21:00:35.027 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-04T21:00:35.027 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-04T21:00:35.028 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-04T21:00:35.028 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-04T21:00:35.028 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-04T21:00:35.028 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-04-04T21:00:35.028 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-04-04T21:00:35.028 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-04T21:00:35.028 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-04T21:00:35.028 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-04T21:00:35.028 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-04T21:00:35.028 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-04T21:00:35.028 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-04T21:00:35.028 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-04T21:00:35.028 INFO:tasks.ceph:[osd] osd class default list = * 2024-04-04T21:00:35.029 INFO:tasks.ceph:[osd] osd class load list = * 2024-04-04T21:00:35.029 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-04-04T21:00:35.029 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-04T21:00:35.029 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-04T21:00:35.029 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-04T21:00:35.029 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-04-04T21:00:35.029 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-04T21:00:35.029 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-04T21:00:35.029 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-04-04T21:00:35.029 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-04T21:00:35.029 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-04T21:00:35.029 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-04T21:00:35.029 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-04T21:00:35.030 INFO:tasks.ceph:Setting up mon.a... 2024-04-04T21:00:35.030 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-04T21:00:35.085 INFO:teuthology.orchestra.run.smithi106.stdout:creating /etc/ceph/ceph.keyring 2024-04-04T21:00:35.089 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-04T21:00:35.135 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-04T21:00:35.164 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.106'), ('mon.c', '[v2:172.21.15.106:3301,v1:172.21.15.106:6790]'), ('mon.b', '172.21.15.203')] 2024-04-04T21:00:35.164 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.106,[v2:172.21.15.106:3301,v1:172.21.15.106:6790],172.21.15.203', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-04T21:00:35.165 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:00:35.165 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-04T21:00:35.219 DEBUG:teuthology.orchestra.run.smithi106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.106 --addv c '[v2:172.21.15.106:3301,v1:172.21.15.106:6790]' --add b 172.21.15.203 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-04T21:00:35.304 INFO:teuthology.orchestra.run.smithi106.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-04T21:00:35.304 INFO:teuthology.orchestra.run.smithi106.stdout:monmaptool: generated fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:00:35.304 INFO:teuthology.orchestra.run.smithi106.stdout:setting min_mon_release = quincy 2024-04-04T21:00:35.304 INFO:teuthology.orchestra.run.smithi106.stdout:epoch 0 2024-04-04T21:00:35.304 INFO:teuthology.orchestra.run.smithi106.stdout:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:00:35.304 INFO:teuthology.orchestra.run.smithi106.stdout:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:00:35.305 INFO:teuthology.orchestra.run.smithi106.stdout:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:00:35.305 INFO:teuthology.orchestra.run.smithi106.stdout:min_mon_release 17 (quincy) 2024-04-04T21:00:35.305 INFO:teuthology.orchestra.run.smithi106.stdout:election_strategy: 1 2024-04-04T21:00:35.305 INFO:teuthology.orchestra.run.smithi106.stdout:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:00:35.305 INFO:teuthology.orchestra.run.smithi106.stdout:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:00:35.305 INFO:teuthology.orchestra.run.smithi106.stdout:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:00:35.305 INFO:teuthology.orchestra.run.smithi106.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-04T21:00:35.306 DEBUG:teuthology.orchestra.run.smithi106:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-04T21:00:35.361 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8f89ce43-1bcb-41cf-9555-c40eeb258155... 2024-04-04T21:00:35.362 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-04T21:00:35.404 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-04T21:00:35.451 INFO:teuthology.orchestra.run.smithi106.stdout:[global] 2024-04-04T21:00:35.451 INFO:teuthology.orchestra.run.smithi106.stdout: chdir = "" 2024-04-04T21:00:35.451 INFO:teuthology.orchestra.run.smithi106.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-04T21:00:35.451 INFO:teuthology.orchestra.run.smithi106.stdout: auth supported = cephx 2024-04-04T21:00:35.451 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.452 INFO:teuthology.orchestra.run.smithi106.stdout: filestore xattr use omap = true 2024-04-04T21:00:35.452 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.452 INFO:teuthology.orchestra.run.smithi106.stdout: mon clock drift allowed = 1.000 2024-04-04T21:00:35.452 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.452 INFO:teuthology.orchestra.run.smithi106.stdout: osd crush chooseleaf type = 0 2024-04-04T21:00:35.452 INFO:teuthology.orchestra.run.smithi106.stdout: auth debug = true 2024-04-04T21:00:35.452 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.452 INFO:teuthology.orchestra.run.smithi106.stdout: ms die on old message = true 2024-04-04T21:00:35.452 INFO:teuthology.orchestra.run.smithi106.stdout: ms die on bug = true 2024-04-04T21:00:35.452 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.452 INFO:teuthology.orchestra.run.smithi106.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-04T21:00:35.452 INFO:teuthology.orchestra.run.smithi106.stdout: mon pg warn max object skew = 0 2024-04-04T21:00:35.452 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.452 INFO:teuthology.orchestra.run.smithi106.stdout: # disable pg_autoscaler by default for new pools 2024-04-04T21:00:35.453 INFO:teuthology.orchestra.run.smithi106.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-04T21:00:35.453 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.453 INFO:teuthology.orchestra.run.smithi106.stdout: osd pool default size = 2 2024-04-04T21:00:35.453 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.453 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd allow primary affinity = true 2024-04-04T21:00:35.453 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd allow pg remap = true 2024-04-04T21:00:35.453 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on legacy crush tunables = false 2024-04-04T21:00:35.453 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on crush straw calc version zero = false 2024-04-04T21:00:35.453 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on no sortbitwise = false 2024-04-04T21:00:35.453 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on osd down out interval zero = false 2024-04-04T21:00:35.453 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on too few osds = false 2024-04-04T21:00:35.453 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-04T21:00:35.453 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-04T21:00:35.453 INFO:teuthology.orchestra.run.smithi106.stdout: mon_allow_pool_size_one = true 2024-04-04T21:00:35.454 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.454 INFO:teuthology.orchestra.run.smithi106.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-04T21:00:35.454 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.454 INFO:teuthology.orchestra.run.smithi106.stdout: osd default data pool replay window = 5 2024-04-04T21:00:35.454 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.454 INFO:teuthology.orchestra.run.smithi106.stdout: mon allow pool delete = true 2024-04-04T21:00:35.454 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.454 INFO:teuthology.orchestra.run.smithi106.stdout: mon cluster log file level = debug 2024-04-04T21:00:35.454 INFO:teuthology.orchestra.run.smithi106.stdout: debug asserts on shutdown = true 2024-04-04T21:00:35.454 INFO:teuthology.orchestra.run.smithi106.stdout: mon health detail to clog = false 2024-04-04T21:00:35.454 INFO:teuthology.orchestra.run.smithi106.stdout: mon host = "172.21.15.106,[v2:172.21.15.106:3301,v1:172.21.15.106:6790],172.21.15.203" 2024-04-04T21:00:35.454 INFO:teuthology.orchestra.run.smithi106.stdout: mon client directed command retry = 5 2024-04-04T21:00:35.454 INFO:teuthology.orchestra.run.smithi106.stdout: mon election default strategy = 1 2024-04-04T21:00:35.454 INFO:teuthology.orchestra.run.smithi106.stdout: ms inject socket failures = 5000 2024-04-04T21:00:35.455 INFO:teuthology.orchestra.run.smithi106.stdout: ms type = async 2024-04-04T21:00:35.455 INFO:teuthology.orchestra.run.smithi106.stdout: osd heartbeat grace = 80 2024-04-04T21:00:35.455 INFO:teuthology.orchestra.run.smithi106.stdout: fsid = 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:00:35.455 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.455 INFO:teuthology.orchestra.run.smithi106.stdout:[osd] 2024-04-04T21:00:35.455 INFO:teuthology.orchestra.run.smithi106.stdout: osd journal size = 100 2024-04-04T21:00:35.455 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.455 INFO:teuthology.orchestra.run.smithi106.stdout: osd scrub load threshold = 5.0 2024-04-04T21:00:35.455 INFO:teuthology.orchestra.run.smithi106.stdout: osd scrub max interval = 600 2024-04-04T21:00:35.455 INFO:teuthology.orchestra.run.smithi106.stdout: osd mclock profile = high_recovery_ops 2024-04-04T21:00:35.455 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.455 INFO:teuthology.orchestra.run.smithi106.stdout: osd recover clone overlap = true 2024-04-04T21:00:35.455 INFO:teuthology.orchestra.run.smithi106.stdout: osd recovery max chunk = 1048576 2024-04-04T21:00:35.455 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.456 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug shutdown = true 2024-04-04T21:00:35.456 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug op order = true 2024-04-04T21:00:35.456 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug verify stray on activate = true 2024-04-04T21:00:35.456 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug trim objects = true 2024-04-04T21:00:35.456 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.456 INFO:teuthology.orchestra.run.smithi106.stdout: osd open classes on start = true 2024-04-04T21:00:35.456 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug pg log writeout = true 2024-04-04T21:00:35.456 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.457 INFO:teuthology.orchestra.run.smithi106.stdout: osd deep scrub update digest min age = 30 2024-04-04T21:00:35.457 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.457 INFO:teuthology.orchestra.run.smithi106.stdout: osd map max advance = 10 2024-04-04T21:00:35.457 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.457 INFO:teuthology.orchestra.run.smithi106.stdout: journal zero on create = true 2024-04-04T21:00:35.457 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.457 INFO:teuthology.orchestra.run.smithi106.stdout: filestore ondisk finisher threads = 3 2024-04-04T21:00:35.457 INFO:teuthology.orchestra.run.smithi106.stdout: filestore apply finisher threads = 3 2024-04-04T21:00:35.457 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.457 INFO:teuthology.orchestra.run.smithi106.stdout: bdev debug aio = true 2024-04-04T21:00:35.457 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug misdirected ops = true 2024-04-04T21:00:35.457 INFO:teuthology.orchestra.run.smithi106.stdout: bdev async discard = True 2024-04-04T21:00:35.457 INFO:teuthology.orchestra.run.smithi106.stdout: bdev enable discard = True 2024-04-04T21:00:35.457 INFO:teuthology.orchestra.run.smithi106.stdout: bluestore block size = 96636764160 2024-04-04T21:00:35.458 INFO:teuthology.orchestra.run.smithi106.stdout: bluestore fsck on mount = True 2024-04-04T21:00:35.458 INFO:teuthology.orchestra.run.smithi106.stdout: bluestore zero block detection = True 2024-04-04T21:00:35.458 INFO:teuthology.orchestra.run.smithi106.stdout: debug bluefs = 1 2024-04-04T21:00:35.458 INFO:teuthology.orchestra.run.smithi106.stdout: debug bluestore = 1 2024-04-04T21:00:35.458 INFO:teuthology.orchestra.run.smithi106.stdout: debug monc = 20 2024-04-04T21:00:35.458 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-04-04T21:00:35.458 INFO:teuthology.orchestra.run.smithi106.stdout: debug osd = 20 2024-04-04T21:00:35.458 INFO:teuthology.orchestra.run.smithi106.stdout: debug rocksdb = 10 2024-04-04T21:00:35.458 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd backfillfull_ratio = 0.85 2024-04-04T21:00:35.458 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd full ratio = 0.9 2024-04-04T21:00:35.458 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd nearfull ratio = 0.8 2024-04-04T21:00:35.458 INFO:teuthology.orchestra.run.smithi106.stdout: osd class default list = * 2024-04-04T21:00:35.458 INFO:teuthology.orchestra.run.smithi106.stdout: osd class load list = * 2024-04-04T21:00:35.458 INFO:teuthology.orchestra.run.smithi106.stdout: osd client watch timeout = 120 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug verify cached snaps = True 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug verify missing on start = True 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout: osd failsafe full ratio = 0.95 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout: osd fast shutdown = False 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout: osd mclock override recovery settings = True 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout: osd memory target = 1610612736 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout: osd objectstore = bluestore 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout: osd op queue = debug_random 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout: osd op queue cut off = debug_random 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout: osd shutdown pgref assert = True 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout:[mgr] 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout: debug mgr = 20 2024-04-04T21:00:35.459 INFO:teuthology.orchestra.run.smithi106.stdout: debug mon = 20 2024-04-04T21:00:35.460 INFO:teuthology.orchestra.run.smithi106.stdout: debug auth = 20 2024-04-04T21:00:35.460 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min pgs per osd = 4 2024-04-04T21:00:35.460 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min bytes per osd = 10 2024-04-04T21:00:35.460 INFO:teuthology.orchestra.run.smithi106.stdout: mgr/telemetry/nag = false 2024-04-04T21:00:35.460 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.460 INFO:teuthology.orchestra.run.smithi106.stdout:[mon] 2024-04-04T21:00:35.460 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-04-04T21:00:35.460 INFO:teuthology.orchestra.run.smithi106.stdout: debug mon = 20 2024-04-04T21:00:35.460 INFO:teuthology.orchestra.run.smithi106.stdout: debug paxos = 20 2024-04-04T21:00:35.460 INFO:teuthology.orchestra.run.smithi106.stdout: debug auth = 20 2024-04-04T21:00:35.460 INFO:teuthology.orchestra.run.smithi106.stdout: mon data avail warn = 5 2024-04-04T21:00:35.460 INFO:teuthology.orchestra.run.smithi106.stdout: mon mgr mkfs grace = 240 2024-04-04T21:00:35.460 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min pgs per osd = 4 2024-04-04T21:00:35.460 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd reporter subtree level = osd 2024-04-04T21:00:35.461 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd prime pg temp = true 2024-04-04T21:00:35.461 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min bytes per osd = 10 2024-04-04T21:00:35.461 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.461 INFO:teuthology.orchestra.run.smithi106.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-04T21:00:35.461 INFO:teuthology.orchestra.run.smithi106.stdout: auth mon ticket ttl = 660 # 11m 2024-04-04T21:00:35.461 INFO:teuthology.orchestra.run.smithi106.stdout: auth service ticket ttl = 240 # 4m 2024-04-04T21:00:35.461 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.461 INFO:teuthology.orchestra.run.smithi106.stdout: # don't complain about insecure global_id in the test suite 2024-04-04T21:00:35.461 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-04T21:00:35.461 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-04T21:00:35.461 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.461 INFO:teuthology.orchestra.run.smithi106.stdout: # 1m isn't quite enough 2024-04-04T21:00:35.461 INFO:teuthology.orchestra.run.smithi106.stdout: mon_down_mkfs_grace = 2m 2024-04-04T21:00:35.461 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.462 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_filestore_osds = false 2024-04-04T21:00:35.462 INFO:teuthology.orchestra.run.smithi106.stdout: mon min osdmap epochs = 50 2024-04-04T21:00:35.462 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd crush smoke test = False 2024-04-04T21:00:35.462 INFO:teuthology.orchestra.run.smithi106.stdout: mon osdmap full prune interval = 2 2024-04-04T21:00:35.462 INFO:teuthology.orchestra.run.smithi106.stdout: mon osdmap full prune min = 15 2024-04-04T21:00:35.462 INFO:teuthology.orchestra.run.smithi106.stdout: mon osdmap full prune txsize = 2 2024-04-04T21:00:35.462 INFO:teuthology.orchestra.run.smithi106.stdout: mon scrub interval = 300 2024-04-04T21:00:35.462 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on pool no app = False 2024-04-04T21:00:35.462 INFO:teuthology.orchestra.run.smithi106.stdout: paxos service trim min = 10 2024-04-04T21:00:35.462 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:00:35.462 INFO:teuthology.orchestra.run.smithi106.stdout:[client] 2024-04-04T21:00:35.462 INFO:teuthology.orchestra.run.smithi106.stdout: rgw cache enabled = true 2024-04-04T21:00:35.462 INFO:teuthology.orchestra.run.smithi106.stdout: rgw enable ops log = true 2024-04-04T21:00:35.462 INFO:teuthology.orchestra.run.smithi106.stdout: rgw enable usage log = true 2024-04-04T21:00:35.463 INFO:teuthology.orchestra.run.smithi106.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-04T21:00:35.463 INFO:teuthology.orchestra.run.smithi106.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-04T21:00:35.463 INFO:teuthology.orchestra.run.smithi106.stdout: debug monc = 20 2024-04-04T21:00:35.463 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-04-04T21:00:35.463 INFO:teuthology.orchestra.run.smithi106.stdout: debug objecter = 20 2024-04-04T21:00:35.464 INFO:teuthology.orchestra.run.smithi106.stdout: debug rados = 20 2024-04-04T21:00:35.464 INFO:teuthology.orchestra.run.smithi106.stdout:[mon.a] 2024-04-04T21:00:35.464 INFO:teuthology.orchestra.run.smithi106.stdout:[mon.c] 2024-04-04T21:00:35.464 INFO:teuthology.orchestra.run.smithi106.stdout:[mon.b] 2024-04-04T21:00:35.465 INFO:teuthology.orchestra.run.smithi203.stdout:[global] 2024-04-04T21:00:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: chdir = "" 2024-04-04T21:00:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-04T21:00:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: auth supported = cephx 2024-04-04T21:00:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: filestore xattr use omap = true 2024-04-04T21:00:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: mon clock drift allowed = 1.000 2024-04-04T21:00:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: osd crush chooseleaf type = 0 2024-04-04T21:00:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: auth debug = true 2024-04-04T21:00:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on old message = true 2024-04-04T21:00:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on bug = true 2024-04-04T21:00:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-04T21:00:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: mon pg warn max object skew = 0 2024-04-04T21:00:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: # disable pg_autoscaler by default for new pools 2024-04-04T21:00:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-04T21:00:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default size = 2 2024-04-04T21:00:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow primary affinity = true 2024-04-04T21:00:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow pg remap = true 2024-04-04T21:00:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on legacy crush tunables = false 2024-04-04T21:00:35.467 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on crush straw calc version zero = false 2024-04-04T21:00:35.467 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on no sortbitwise = false 2024-04-04T21:00:35.467 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on osd down out interval zero = false 2024-04-04T21:00:35.467 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on too few osds = false 2024-04-04T21:00:35.467 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-04T21:00:35.467 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-04T21:00:35.467 INFO:teuthology.orchestra.run.smithi203.stdout: mon_allow_pool_size_one = true 2024-04-04T21:00:35.467 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.467 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-04T21:00:35.467 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.467 INFO:teuthology.orchestra.run.smithi203.stdout: osd default data pool replay window = 5 2024-04-04T21:00:35.467 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.467 INFO:teuthology.orchestra.run.smithi203.stdout: mon allow pool delete = true 2024-04-04T21:00:35.467 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.468 INFO:teuthology.orchestra.run.smithi203.stdout: mon cluster log file level = debug 2024-04-04T21:00:35.468 INFO:teuthology.orchestra.run.smithi203.stdout: debug asserts on shutdown = true 2024-04-04T21:00:35.468 INFO:teuthology.orchestra.run.smithi203.stdout: mon health detail to clog = false 2024-04-04T21:00:35.468 INFO:teuthology.orchestra.run.smithi203.stdout: mon host = "172.21.15.106,[v2:172.21.15.106:3301,v1:172.21.15.106:6790],172.21.15.203" 2024-04-04T21:00:35.468 INFO:teuthology.orchestra.run.smithi203.stdout: mon client directed command retry = 5 2024-04-04T21:00:35.468 INFO:teuthology.orchestra.run.smithi203.stdout: mon election default strategy = 1 2024-04-04T21:00:35.468 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject socket failures = 5000 2024-04-04T21:00:35.468 INFO:teuthology.orchestra.run.smithi203.stdout: ms type = async 2024-04-04T21:00:35.468 INFO:teuthology.orchestra.run.smithi203.stdout: osd heartbeat grace = 80 2024-04-04T21:00:35.468 INFO:teuthology.orchestra.run.smithi203.stdout: fsid = 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:00:35.468 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.468 INFO:teuthology.orchestra.run.smithi203.stdout:[osd] 2024-04-04T21:00:35.468 INFO:teuthology.orchestra.run.smithi203.stdout: osd journal size = 100 2024-04-04T21:00:35.468 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub load threshold = 5.0 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub max interval = 600 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock profile = high_recovery_ops 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: osd recover clone overlap = true 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: osd recovery max chunk = 1048576 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug shutdown = true 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug op order = true 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify stray on activate = true 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug trim objects = true 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: osd open classes on start = true 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug pg log writeout = true 2024-04-04T21:00:35.469 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.470 INFO:teuthology.orchestra.run.smithi203.stdout: osd deep scrub update digest min age = 30 2024-04-04T21:00:35.470 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.470 INFO:teuthology.orchestra.run.smithi203.stdout: osd map max advance = 10 2024-04-04T21:00:35.470 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.470 INFO:teuthology.orchestra.run.smithi203.stdout: journal zero on create = true 2024-04-04T21:00:35.470 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.470 INFO:teuthology.orchestra.run.smithi203.stdout: filestore ondisk finisher threads = 3 2024-04-04T21:00:35.470 INFO:teuthology.orchestra.run.smithi203.stdout: filestore apply finisher threads = 3 2024-04-04T21:00:35.470 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.470 INFO:teuthology.orchestra.run.smithi203.stdout: bdev debug aio = true 2024-04-04T21:00:35.470 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug misdirected ops = true 2024-04-04T21:00:35.470 INFO:teuthology.orchestra.run.smithi203.stdout: bdev async discard = True 2024-04-04T21:00:35.470 INFO:teuthology.orchestra.run.smithi203.stdout: bdev enable discard = True 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore block size = 96636764160 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore fsck on mount = True 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore zero block detection = True 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluefs = 1 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluestore = 1 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 20 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: debug osd = 20 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: debug rocksdb = 10 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd backfillfull_ratio = 0.85 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd full ratio = 0.9 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd nearfull ratio = 0.8 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: osd class default list = * 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: osd class load list = * 2024-04-04T21:00:35.471 INFO:teuthology.orchestra.run.smithi203.stdout: osd client watch timeout = 120 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify cached snaps = True 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify missing on start = True 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout: osd failsafe full ratio = 0.95 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout: osd fast shutdown = False 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock override recovery settings = True 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout: osd memory target = 1610612736 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout: osd objectstore = bluestore 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue = debug_random 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue cut off = debug_random 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout: osd shutdown pgref assert = True 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr] 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout: debug mgr = 20 2024-04-04T21:00:35.472 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout: mgr/telemetry/nag = false 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout:[mon] 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout: debug paxos = 20 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout: mon data avail warn = 5 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout: mon mgr mkfs grace = 240 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd reporter subtree level = osd 2024-04-04T21:00:35.473 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd prime pg temp = true 2024-04-04T21:00:35.474 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-04-04T21:00:35.474 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.474 INFO:teuthology.orchestra.run.smithi203.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-04T21:00:35.474 INFO:teuthology.orchestra.run.smithi203.stdout: auth mon ticket ttl = 660 # 11m 2024-04-04T21:00:35.474 INFO:teuthology.orchestra.run.smithi203.stdout: auth service ticket ttl = 240 # 4m 2024-04-04T21:00:35.474 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.474 INFO:teuthology.orchestra.run.smithi203.stdout: # don't complain about insecure global_id in the test suite 2024-04-04T21:00:35.474 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-04T21:00:35.474 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-04T21:00:35.474 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.474 INFO:teuthology.orchestra.run.smithi203.stdout: # 1m isn't quite enough 2024-04-04T21:00:35.474 INFO:teuthology.orchestra.run.smithi203.stdout: mon_down_mkfs_grace = 2m 2024-04-04T21:00:35.474 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.474 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_filestore_osds = false 2024-04-04T21:00:35.475 INFO:teuthology.orchestra.run.smithi203.stdout: mon min osdmap epochs = 50 2024-04-04T21:00:35.475 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd crush smoke test = False 2024-04-04T21:00:35.475 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune interval = 2 2024-04-04T21:00:35.475 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune min = 15 2024-04-04T21:00:35.475 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune txsize = 2 2024-04-04T21:00:35.475 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub interval = 300 2024-04-04T21:00:35.475 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on pool no app = False 2024-04-04T21:00:35.475 INFO:teuthology.orchestra.run.smithi203.stdout: paxos service trim min = 10 2024-04-04T21:00:35.475 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-04T21:00:35.475 INFO:teuthology.orchestra.run.smithi203.stdout:[client] 2024-04-04T21:00:35.475 INFO:teuthology.orchestra.run.smithi203.stdout: rgw cache enabled = true 2024-04-04T21:00:35.475 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable ops log = true 2024-04-04T21:00:35.475 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable usage log = true 2024-04-04T21:00:35.475 INFO:teuthology.orchestra.run.smithi203.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-04T21:00:35.476 INFO:teuthology.orchestra.run.smithi203.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-04T21:00:35.476 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 20 2024-04-04T21:00:35.476 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-04-04T21:00:35.476 INFO:teuthology.orchestra.run.smithi203.stdout: debug objecter = 20 2024-04-04T21:00:35.476 INFO:teuthology.orchestra.run.smithi203.stdout: debug rados = 20 2024-04-04T21:00:35.476 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.a] 2024-04-04T21:00:35.476 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.c] 2024-04-04T21:00:35.476 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.b] 2024-04-04T21:00:35.477 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-04T21:00:35.478 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-04T21:00:35.572 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-04T21:00:35.573 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:00:35.573 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-04T21:00:35.588 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:00:35.588 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-04T21:00:35.643 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi106.front.sepia.ceph.com 2024-04-04T21:00:35.643 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:00:35.643 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-04T21:00:35.643 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-04T21:00:35.723 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:00:35.723 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-04T21:00:35.778 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi203.front.sepia.ceph.com 2024-04-04T21:00:35.778 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:00:35.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-04T21:00:35.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-04T21:00:35.821 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:00:35.821 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-04T21:00:35.877 INFO:tasks.ceph:Setting up mon nodes... 2024-04-04T21:00:35.877 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-04T21:00:35.878 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-04T21:00:35.932 INFO:teuthology.orchestra.run.smithi106.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-04T21:00:35.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-04T21:00:35.998 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-04T21:00:36.001 INFO:tasks.ceph:Setting up mds nodes... 2024-04-04T21:00:36.001 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-04T21:00:36.001 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-04T21:00:36.044 INFO:teuthology.orchestra.run.smithi106.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-04T21:00:36.059 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-04T21:00:36.103 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-04T21:00:36.118 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-04T21:00:36.118 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi106.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-04T21:00:36.119 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-04T21:00:36.145 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-04T21:00:36.145 INFO:tasks.ceph:role: osd.0 2024-04-04T21:00:36.145 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi106.front.sepia.ceph.com 2024-04-04T21:00:36.146 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-04T21:00:36.219 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-04T21:00:36.219 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-04T21:00:36.219 INFO:teuthology.orchestra.run.smithi106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-04T21:00:36.220 INFO:teuthology.orchestra.run.smithi106.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-04T21:00:36.220 INFO:teuthology.orchestra.run.smithi106.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-04T21:00:36.220 INFO:teuthology.orchestra.run.smithi106.stdout: = sunit=0 swidth=0 blks 2024-04-04T21:00:36.220 INFO:teuthology.orchestra.run.smithi106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-04T21:00:36.220 INFO:teuthology.orchestra.run.smithi106.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-04T21:00:36.220 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-04T21:00:36.220 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-04T21:00:36.722 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done. 2024-04-04T21:00:36.724 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi106.front.sepia.ceph.com -o noatime 2024-04-04T21:00:36.724 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-04T21:00:36.868 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-04T21:00:36.902 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-04T21:00:36.970 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-04T21:00:36.970 INFO:tasks.ceph:role: osd.1 2024-04-04T21:00:36.970 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi106.front.sepia.ceph.com 2024-04-04T21:00:36.970 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-04T21:00:37.044 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-04T21:00:37.044 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-04T21:00:37.044 INFO:teuthology.orchestra.run.smithi106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-04T21:00:37.044 INFO:teuthology.orchestra.run.smithi106.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-04T21:00:37.044 INFO:teuthology.orchestra.run.smithi106.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-04T21:00:37.044 INFO:teuthology.orchestra.run.smithi106.stdout: = sunit=0 swidth=0 blks 2024-04-04T21:00:37.045 INFO:teuthology.orchestra.run.smithi106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-04T21:00:37.045 INFO:teuthology.orchestra.run.smithi106.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-04T21:00:37.045 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-04T21:00:37.045 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-04T21:00:37.498 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done. 2024-04-04T21:00:37.500 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi106.front.sepia.ceph.com -o noatime 2024-04-04T21:00:37.500 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-04T21:00:37.596 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-04T21:00:37.628 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-04T21:00:37.696 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-04T21:00:37.696 INFO:tasks.ceph:role: osd.2 2024-04-04T21:00:37.696 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi106.front.sepia.ceph.com 2024-04-04T21:00:37.696 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-04T21:00:37.770 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-04T21:00:37.771 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-04T21:00:37.771 INFO:teuthology.orchestra.run.smithi106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-04T21:00:37.771 INFO:teuthology.orchestra.run.smithi106.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-04T21:00:37.771 INFO:teuthology.orchestra.run.smithi106.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-04T21:00:37.771 INFO:teuthology.orchestra.run.smithi106.stdout: = sunit=0 swidth=0 blks 2024-04-04T21:00:37.771 INFO:teuthology.orchestra.run.smithi106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-04T21:00:37.771 INFO:teuthology.orchestra.run.smithi106.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-04T21:00:37.771 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-04T21:00:37.771 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-04T21:00:38.263 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done. 2024-04-04T21:00:38.265 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi106.front.sepia.ceph.com -o noatime 2024-04-04T21:00:38.265 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-04T21:00:38.413 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-04T21:00:38.449 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-04T21:00:38.518 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-04T21:00:38.518 INFO:tasks.ceph:role: osd.3 2024-04-04T21:00:38.518 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi106.front.sepia.ceph.com 2024-04-04T21:00:38.519 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-04T21:00:38.594 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-04T21:00:38.594 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-04T21:00:38.594 INFO:teuthology.orchestra.run.smithi106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-04T21:00:38.594 INFO:teuthology.orchestra.run.smithi106.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-04T21:00:38.594 INFO:teuthology.orchestra.run.smithi106.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-04T21:00:38.594 INFO:teuthology.orchestra.run.smithi106.stdout: = sunit=0 swidth=0 blks 2024-04-04T21:00:38.595 INFO:teuthology.orchestra.run.smithi106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-04T21:00:38.595 INFO:teuthology.orchestra.run.smithi106.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-04T21:00:38.595 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-04T21:00:38.595 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-04T21:00:39.109 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done. 2024-04-04T21:00:39.111 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi106.front.sepia.ceph.com -o noatime 2024-04-04T21:00:39.111 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-04T21:00:39.247 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-04T21:00:39.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-04T21:00:39.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:39.397+0000 7f7a94ed3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-04T21:00:39.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:39.397+0000 7f7a94ed3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-04T21:00:39.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:39.397+0000 7f7a94ed3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-04T21:00:39.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:39.397+0000 7f7a94ed3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-04T21:00:39.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:39.397+0000 7f7a94ed3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-04T21:00:39.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:39.398+0000 7f7a94ed3640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-04T21:00:42.245 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-04T21:00:42.278 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-04T21:00:42.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:42.373+0000 7f4236c14640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-04T21:00:42.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:42.373+0000 7f4236c14640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-04T21:00:42.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:42.373+0000 7f4236c14640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-04T21:00:42.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:42.373+0000 7f4236c14640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-04T21:00:42.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:42.373+0000 7f4236c14640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-04T21:00:42.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:42.374+0000 7f4236c14640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-04T21:00:45.228 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-04T21:00:45.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-04T21:00:45.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:45.356+0000 7f675852a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-04T21:00:45.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:45.356+0000 7f675852a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-04T21:00:45.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:45.356+0000 7f675852a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-04T21:00:45.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:45.356+0000 7f675852a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-04T21:00:45.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:45.356+0000 7f675852a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-04T21:00:45.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:45.357+0000 7f675852a640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-04T21:00:48.208 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-04T21:00:48.240 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-04T21:00:48.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:48.334+0000 7fc5a0a20640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-04T21:00:48.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:48.335+0000 7fc5a0a20640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-04T21:00:48.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:48.335+0000 7fc5a0a20640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-04T21:00:48.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:48.335+0000 7fc5a0a20640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-04T21:00:48.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:48.335+0000 7fc5a0a20640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-04T21:00:48.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:00:48.335+0000 7fc5a0a20640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-04T21:00:51.181 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-04T21:00:51.214 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-04T21:00:51.246 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-04T21:00:51.246 INFO:tasks.ceph:role: osd.4 2024-04-04T21:00:51.246 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-04T21:00:51.246 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-04T21:00:51.325 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-04T21:00:51.325 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-04T21:00:51.325 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-04T21:00:51.325 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-04T21:00:51.325 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-04T21:00:51.325 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-04-04T21:00:51.326 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-04T21:00:51.326 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-04T21:00:51.326 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-04T21:00:51.326 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-04T21:00:51.847 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-04-04T21:00:51.849 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-04-04T21:00:51.849 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-04T21:00:51.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-04T21:00:52.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-04T21:00:52.084 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-04T21:00:52.084 INFO:tasks.ceph:role: osd.5 2024-04-04T21:00:52.084 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-04T21:00:52.084 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-04T21:00:52.161 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-04T21:00:52.161 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-04T21:00:52.161 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-04T21:00:52.161 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-04T21:00:52.161 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-04T21:00:52.161 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-04-04T21:00:52.161 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-04T21:00:52.161 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-04T21:00:52.161 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-04T21:00:52.162 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-04T21:00:52.620 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-04-04T21:00:52.622 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-04-04T21:00:52.622 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-04T21:00:52.720 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-04T21:00:52.753 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-04T21:00:52.822 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-04T21:00:52.822 INFO:tasks.ceph:role: osd.6 2024-04-04T21:00:52.823 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-04T21:00:52.823 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-04T21:00:52.899 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-04T21:00:52.899 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-04T21:00:52.899 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-04T21:00:52.899 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-04T21:00:52.899 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-04T21:00:52.900 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-04-04T21:00:52.900 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-04T21:00:52.900 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-04T21:00:52.900 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-04T21:00:52.900 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-04T21:00:53.394 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-04-04T21:00:53.396 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-04-04T21:00:53.397 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-04T21:00:53.555 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-04T21:00:53.590 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-04T21:00:53.660 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-04T21:00:53.660 INFO:tasks.ceph:role: osd.7 2024-04-04T21:00:53.660 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-04T21:00:53.661 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-04T21:00:53.736 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-04T21:00:53.736 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-04T21:00:53.736 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-04T21:00:53.736 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-04T21:00:53.736 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-04T21:00:53.736 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-04-04T21:00:53.736 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-04T21:00:53.736 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-04T21:00:53.736 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-04T21:00:53.736 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-04T21:00:54.236 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-04-04T21:00:54.239 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-04-04T21:00:54.239 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-04T21:00:54.369 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-04T21:00:54.406 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-04T21:00:54.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:00:54.510+0000 7f88c1111640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-04T21:00:54.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:00:54.510+0000 7f88c1111640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-04T21:00:54.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:00:54.510+0000 7f88c1111640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-04T21:00:54.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:00:54.510+0000 7f88c1111640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-04T21:00:54.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:00:54.510+0000 7f88c1111640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-04T21:00:54.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:00:54.511+0000 7f88c1111640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-04T21:00:57.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-04T21:00:57.399 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-04T21:00:57.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:00:57.491+0000 7f5c8ad49640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-04T21:00:57.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:00:57.491+0000 7f5c8ad49640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-04T21:00:57.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:00:57.491+0000 7f5c8ad49640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-04T21:00:57.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:00:57.491+0000 7f5c8ad49640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-04T21:00:57.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:00:57.491+0000 7f5c8ad49640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-04T21:00:57.492 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:00:57.491+0000 7f5c8ad49640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-04T21:01:00.333 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-04T21:01:00.364 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-04T21:01:00.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:00.455+0000 7f8a2ec0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-04T21:01:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:00.455+0000 7f8a2ec0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-04T21:01:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:00.455+0000 7f8a2ec0c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-04T21:01:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:00.455+0000 7f8a2ec0c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-04T21:01:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:00.455+0000 7f8a2ec0c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-04T21:01:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:00.456+0000 7f8a2ec0c640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-04T21:01:03.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-04T21:01:03.333 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-04T21:01:03.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:03.424+0000 7f1b0700d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-04T21:01:03.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:03.424+0000 7f1b0700d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-04T21:01:03.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:03.424+0000 7f1b0700d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-04T21:01:03.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:03.424+0000 7f1b0700d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-04T21:01:03.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:03.424+0000 7f1b0700d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-04T21:01:03.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:03.425+0000 7f1b0700d640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-04T21:01:06.267 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-04T21:01:06.299 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-04T21:01:06.300 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:01:06.300 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-04T21:01:06.335 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:01:06.335 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-04T21:01:06.406 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:01:06.406 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-04T21:01:06.476 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:01:06.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-04T21:01:06.545 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:01:06.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-04T21:01:06.619 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:01:06.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-04T21:01:06.652 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:01:06.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-04T21:01:06.723 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:01:06.723 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-04T21:01:06.792 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:01:06.793 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-04T21:01:06.861 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:01:06.862 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-04T21:01:06.931 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:01:06.931 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-04T21:01:06.948 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:01:06.948 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-04T21:01:06.986 INFO:tasks.ceph:Adding keys to all mons... 2024-04-04T21:01:06.987 DEBUG:teuthology.orchestra.run.smithi106:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-04T21:01:06.991 DEBUG:teuthology.orchestra.run.smithi203:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-04T21:01:07.033 INFO:teuthology.orchestra.run.smithi106.stdout:[mgr.y] 2024-04-04T21:01:07.033 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQDzFA9mN7eYNxAA2PT2hSZEOMUKEnDmVlmKng== 2024-04-04T21:01:07.033 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.0] 2024-04-04T21:01:07.033 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQD3FA9mxCS6FxAA09pIkoz/FEcPVpyQDoirtg== 2024-04-04T21:01:07.033 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.1] 2024-04-04T21:01:07.034 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQD6FA9mhOJUFhAA5a0PtUtbPG9u/BsHSLBn2g== 2024-04-04T21:01:07.034 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.2] 2024-04-04T21:01:07.034 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQD9FA9mKydOFRAApOayR0mkkpQxF/DJ2S5Dag== 2024-04-04T21:01:07.034 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.3] 2024-04-04T21:01:07.034 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQAAFQ9mpMIEFBAA0T9MyfRiDYICUdvCi+/T5g== 2024-04-04T21:01:07.035 INFO:teuthology.orchestra.run.smithi106.stdout:[mgr.x] 2024-04-04T21:01:07.035 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQDzFA9mqoKQOxAAy7hbkkvuaQn0N0pX/IBPUw== 2024-04-04T21:01:07.035 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.4] 2024-04-04T21:01:07.035 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQAGFQ9mECt0HhAALs9W9NeGSazCVazQO6QPdQ== 2024-04-04T21:01:07.036 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.5] 2024-04-04T21:01:07.037 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQAJFQ9mxUtOHRAAGATR2YnI68JZnP7v6DiL1A== 2024-04-04T21:01:07.037 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.6] 2024-04-04T21:01:07.037 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQAMFQ9mmEMsGxAA+2obFgTTY6vwxxO0tPFIHQ== 2024-04-04T21:01:07.037 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.7] 2024-04-04T21:01:07.038 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQAPFQ9mjHhYGRAA8cgeT6nNyfLytwkzIpGd2w== 2024-04-04T21:01:07.038 INFO:teuthology.orchestra.run.smithi106.stdout:[client.0] 2024-04-04T21:01:07.038 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQD0FA9mjY+rAhAAxGeGd0aYAgIpcw3f/PS+Ig== 2024-04-04T21:01:07.038 INFO:teuthology.orchestra.run.smithi106.stdout:[client.1] 2024-04-04T21:01:07.039 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQD0FA9mzdkxBhAAErijW/+vEErQIny/OmpvlQ== 2024-04-04T21:01:07.054 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.y] 2024-04-04T21:01:07.057 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDzFA9mN7eYNxAA2PT2hSZEOMUKEnDmVlmKng== 2024-04-04T21:01:07.057 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.0] 2024-04-04T21:01:07.057 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQD3FA9mxCS6FxAA09pIkoz/FEcPVpyQDoirtg== 2024-04-04T21:01:07.057 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.1] 2024-04-04T21:01:07.057 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQD6FA9mhOJUFhAA5a0PtUtbPG9u/BsHSLBn2g== 2024-04-04T21:01:07.057 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.2] 2024-04-04T21:01:07.057 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQD9FA9mKydOFRAApOayR0mkkpQxF/DJ2S5Dag== 2024-04-04T21:01:07.057 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.3] 2024-04-04T21:01:07.057 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAAFQ9mpMIEFBAA0T9MyfRiDYICUdvCi+/T5g== 2024-04-04T21:01:07.058 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.x] 2024-04-04T21:01:07.058 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDzFA9mqoKQOxAAy7hbkkvuaQn0N0pX/IBPUw== 2024-04-04T21:01:07.058 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.4] 2024-04-04T21:01:07.058 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAGFQ9mECt0HhAALs9W9NeGSazCVazQO6QPdQ== 2024-04-04T21:01:07.058 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.5] 2024-04-04T21:01:07.058 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAJFQ9mxUtOHRAAGATR2YnI68JZnP7v6DiL1A== 2024-04-04T21:01:07.058 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.6] 2024-04-04T21:01:07.058 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAMFQ9mmEMsGxAA+2obFgTTY6vwxxO0tPFIHQ== 2024-04-04T21:01:07.058 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.7] 2024-04-04T21:01:07.058 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAPFQ9mjHhYGRAA8cgeT6nNyfLytwkzIpGd2w== 2024-04-04T21:01:07.058 INFO:teuthology.orchestra.run.smithi203.stdout:[client.0] 2024-04-04T21:01:07.058 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQD0FA9mjY+rAhAAxGeGd0aYAgIpcw3f/PS+Ig== 2024-04-04T21:01:07.059 INFO:teuthology.orchestra.run.smithi203.stdout:[client.1] 2024-04-04T21:01:07.059 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQD0FA9mzdkxBhAAErijW/+vEErQIny/OmpvlQ== 2024-04-04T21:01:07.060 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-04T21:01:07.083 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-04T21:01:07.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.163 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.220 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.272 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.331 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.334 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-04T21:01:07.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-04T21:01:07.440 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.442 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.496 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.548 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.554 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.608 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-04T21:01:07.671 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-04T21:01:07.675 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-04T21:01:07.733 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-04T21:01:07.739 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-04T21:01:07.796 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-04T21:01:07.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-04T21:01:07.826 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-04T21:01:07.925 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-04T21:01:07.959 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-04T21:01:08.030 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-04T21:01:08.129 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-04T21:01:08.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-04T21:01:08.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-04T21:01:08.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-04T21:01:08.316 DEBUG:teuthology.orchestra.run.smithi106:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-04T21:01:08.319 DEBUG:teuthology.orchestra.run.smithi203:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-04T21:01:08.371 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-04T21:01:08.371 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-04-04T21:01:08.371 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-04T21:01:08.372 DEBUG:teuthology.orchestra.run.smithi106:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-04-04T21:01:08.374 INFO:tasks.ceph.mon.a:Started 2024-04-04T21:01:08.374 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-04-04T21:01:08.375 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-04T21:01:08.375 DEBUG:teuthology.orchestra.run.smithi106:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-04-04T21:01:08.377 INFO:tasks.ceph.mon.c:Started 2024-04-04T21:01:08.378 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-04-04T21:01:08.378 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-04T21:01:08.378 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-04-04T21:01:08.413 INFO:tasks.ceph.mon.b:Started 2024-04-04T21:01:08.413 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-04T21:01:08.414 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-04T21:01:08.414 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-04-04T21:01:08.416 INFO:tasks.ceph.mgr.y:Started 2024-04-04T21:01:08.416 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-04T21:01:08.416 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-04-04T21:01:08.418 INFO:tasks.ceph.mgr.x:Started 2024-04-04T21:01:08.418 INFO:tasks.ceph:Setting crush tunables to default 2024-04-04T21:01:08.418 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd crush tunables default 2024-04-04T21:01:08.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.518+0000 7f5f1ec50640 10 monclient: get_monmap_and_config 2024-04-04T21:01:08.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.518+0000 7f5f1ec50640 10 monclient: build_initial_monmap 2024-04-04T21:01:08.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.518+0000 7f5f1ec50640 10 monclient: monmap: 2024-04-04T21:01:08.519 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:01:08.519 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:08.520 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:01:08.519928+0000 2024-04-04T21:01:08.520 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:01:08.519928+0000 2024-04-04T21:01:08.520 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:08.520 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:08.520 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:08.520 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:08.520 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:08.520 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:08.520 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:08.520 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:01:08.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.519+0000 7f5f1ec50640 1 Processor -- start 2024-04-04T21:01:08.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.519+0000 7f5f1ec50640 1 -- start start 2024-04-04T21:01:08.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.519+0000 7f5f1ec50640 10 monclient: init 2024-04-04T21:01:08.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1ec50640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:08.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1ec50640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-04-04T21:01:08.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1ec50640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f180a9b60 addr v1:172.21.15.203:6789/0 2024-04-04T21:01:08.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1ec50640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f181426f0 0x7f5f180b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:08.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1ec50640 10 monclient(hunting): picked mon.noname-b con 0x7f5f181426f0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:08.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1ec50640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5f1813f810 0x7f5f1813fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:08.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1ec50640 10 monclient(hunting): picked mon.noname-c con 0x7f5f1813f810 addr v2:172.21.15.203:3300/0 2024-04-04T21:01:08.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1ec50640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f180b2e00 con 0x7f5f180a9b60 2024-04-04T21:01:08.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1ec50640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:08.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1ec50640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f180a9f40 con 0x7f5f1813f810 2024-04-04T21:01:08.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1ec50640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:08.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1ec50640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f18142c10 con 0x7f5f181426f0 2024-04-04T21:01:08.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1ec50640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:08.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1ec50640 10 monclient(hunting): authenticate will time out at 1317.787535s 2024-04-04T21:01:08.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f17fff640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f181426f0 msgr2=0x7f5f180b28c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-04-04T21:01:08.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f17fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f181426f0 0x7f5f180b28c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-04T21:01:08.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1c9ec640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f5f180a9b60 legacy=0x7f5f181430a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-04T21:01:08.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1d1ed640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f5f1813f810 msgr2=0x7f5f1813fc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-04-04T21:01:08.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.520+0000 7f5f1d1ed640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5f1813f810 0x7f5f1813fc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-04T21:01:08.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.721+0000 7f5f17fff640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f181426f0 msgr2=0x7f5f180b28c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-04-04T21:01:08.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.721+0000 7f5f17fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f181426f0 0x7f5f180b28c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-04T21:01:08.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.721+0000 7f5f1d1ed640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f5f1813f810 msgr2=0x7f5f1813fc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-04-04T21:01:08.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.721+0000 7f5f1c9ec640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f5f180a9b60 legacy=0x7f5f181430a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-04T21:01:08.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:08.721+0000 7f5f1d1ed640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5f1813f810 0x7f5f1813fc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-04T21:01:09.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:09.122+0000 7f5f17fff640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f181426f0 msgr2=0x7f5f180b28c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-04-04T21:01:09.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:09.122+0000 7f5f1d1ed640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f5f1813f810 msgr2=0x7f5f1813fc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-04-04T21:01:09.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:09.122+0000 7f5f1c9ec640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f5f180a9b60 legacy=0x7f5f181430a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-04T21:01:09.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:09.122+0000 7f5f17fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f181426f0 0x7f5f180b28c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-04T21:01:09.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:09.122+0000 7f5f1d1ed640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5f1813f810 0x7f5f1813fc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-04T21:01:09.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:09.923+0000 7f5f17fff640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f181426f0 msgr2=0x7f5f180b28c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-04-04T21:01:09.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:09.923+0000 7f5f17fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f181426f0 0x7f5f180b28c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-04T21:01:09.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:09.923+0000 7f5f1c9ec640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f5f180a9b60 legacy=0x7f5f181430a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-04T21:01:09.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:09.923+0000 7f5f1d1ed640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f5f1813f810 msgr2=0x7f5f1813fc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-04-04T21:01:09.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:09.923+0000 7f5f1d1ed640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5f1813f810 0x7f5f1813fc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-04T21:01:11.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 10 monclient(hunting): tick 2024-04-04T21:01:11.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 10 monclient(hunting): _check_auth_tickets 2024-04-04T21:01:11.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 1 monclient(hunting): continuing hunt 2024-04-04T21:01:11.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-04T21:01:11.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f181426f0 msgr2=0x7f5f180b28c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-04T21:01:11.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f181426f0 0x7f5f180b28c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:11.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f5f1813f810 msgr2=0x7f5f1813fc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-04T21:01:11.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5f1813f810 0x7f5f1813fc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:11.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f5f180a9b60 legacy=0x7f5f181430a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-04T21:01:11.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 10 monclient: _add_conns ranks=[3,0] 2024-04-04T21:01:11.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5efc0036c0 0x7f5efc005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:11.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 10 monclient(hunting): picked mon.noname-a con 0x7f5efc0036c0 addr v2:172.21.15.106:3300/0 2024-04-04T21:01:11.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5efc006190 addr v1:172.21.15.106:6789/0 2024-04-04T21:01:11.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f180b8c90 con 0x7f5efc006190 2024-04-04T21:01:11.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:11.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f180a9f40 con 0x7f5efc0036c0 2024-04-04T21:01:11.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f167fc640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:11.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f17fff640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f5efc006190 legacy=0x7f5efc008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-04-04T21:01:11.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f1c9ec640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f5efc0036c0 msgr2=0x7f5efc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-04-04T21:01:11.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.520+0000 7f5f1c9ec640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5efc0036c0 0x7f5efc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-04T21:01:11.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.721+0000 7f5f17fff640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f5efc006190 legacy=0x7f5efc008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-04-04T21:01:11.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.721+0000 7f5f1c9ec640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f5efc0036c0 msgr2=0x7f5efc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-04-04T21:01:11.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:11.721+0000 7f5f1c9ec640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5efc0036c0 0x7f5efc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-04T21:01:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:12.122+0000 7f5f1c9ec640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f5efc0036c0 msgr2=0x7f5efc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-04-04T21:01:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:12.122+0000 7f5f17fff640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f5efc006190 legacy=0x7f5efc008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-04-04T21:01:12.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:12.122+0000 7f5f1c9ec640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5efc0036c0 0x7f5efc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-04T21:01:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:12.923+0000 7f5f17fff640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f5efc006190 legacy=0x7f5efc008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-04-04T21:01:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:12.923+0000 7f5f1c9ec640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f5efc0036c0 msgr2=0x7f5efc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-04-04T21:01:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:12.923+0000 7f5f1c9ec640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5efc0036c0 0x7f5efc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-04T21:01:14.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 10 monclient(hunting): tick 2024-04-04T21:01:14.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 10 monclient(hunting): _check_auth_tickets 2024-04-04T21:01:14.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 1 monclient(hunting): continuing hunt 2024-04-04T21:01:14.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-04T21:01:14.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f5efc0036c0 msgr2=0x7f5efc005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-04T21:01:14.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5efc0036c0 0x7f5efc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:14.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f5efc006190 legacy=0x7f5efc008650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-04T21:01:14.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f1d1ed640 1 -- reap_dead start 2024-04-04T21:01:14.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-04-04T21:01:14.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5efc009f00 addr v1:172.21.15.106:6789/0 2024-04-04T21:01:14.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5efc00aa90 0x7f5efc005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:14.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 10 monclient(hunting): picked mon.noname-b con 0x7f5efc00aa90 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:14.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5efc0087d0 0x7f5efc00d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:14.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 10 monclient(hunting): picked mon.noname-c con 0x7f5efc0087d0 addr v2:172.21.15.203:3300/0 2024-04-04T21:01:14.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f180b2e00 con 0x7f5efc009f00 2024-04-04T21:01:14.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:14.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f180a9f40 con 0x7f5efc0087d0 2024-04-04T21:01:14.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:14.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f18142c10 con 0x7f5efc00aa90 2024-04-04T21:01:14.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f167fc640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:14.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f1c9ec640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f5efc009f00 legacy=0x7f5efc00a2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-04-04T21:01:14.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f17fff640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5efc00aa90 msgr2=0x7f5efc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-04-04T21:01:14.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f17fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5efc00aa90 0x7f5efc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-04T21:01:14.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f1d1ed640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f5efc0087d0 msgr2=0x7f5efc00d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-04-04T21:01:14.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.520+0000 7f5f1d1ed640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5efc0087d0 0x7f5efc00d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-04T21:01:14.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.721+0000 7f5f17fff640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5efc00aa90 msgr2=0x7f5efc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-04-04T21:01:14.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.721+0000 7f5f1c9ec640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f5efc009f00 legacy=0x7f5efc00a2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-04-04T21:01:14.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.721+0000 7f5f17fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5efc00aa90 0x7f5efc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-04T21:01:14.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.721+0000 7f5f1d1ed640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f5efc0087d0 msgr2=0x7f5efc00d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-04-04T21:01:14.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:14.721+0000 7f5f1d1ed640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5efc0087d0 0x7f5efc00d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-04T21:01:15.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:15.122+0000 7f5f1c9ec640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f5efc009f00 legacy=0x7f5efc00a2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-04-04T21:01:15.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:15.122+0000 7f5f17fff640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5efc00aa90 msgr2=0x7f5efc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-04-04T21:01:15.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:15.122+0000 7f5f17fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5efc00aa90 0x7f5efc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-04T21:01:15.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:15.122+0000 7f5f1d1ed640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f5efc0087d0 msgr2=0x7f5efc00d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-04-04T21:01:15.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:15.122+0000 7f5f1d1ed640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5efc0087d0 0x7f5efc00d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-04T21:01:15.452 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:00:06.934 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:01:15.687 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:00:07.204 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:01:15.712 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:00:07.229 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:01:15.759 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:00:07.241 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:01:15.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:15.923+0000 7f5f17fff640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5efc00aa90 msgr2=0x7f5efc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-04-04T21:01:15.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:15.923+0000 7f5f17fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5efc00aa90 0x7f5efc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-04T21:01:15.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:15.923+0000 7f5f1c9ec640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f5efc009f00 legacy=0x7f5efc00a2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-04-04T21:01:15.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:15.923+0000 7f5f1d1ed640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f5efc0087d0 msgr2=0x7f5efc00d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-04-04T21:01:15.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:15.923+0000 7f5f1d1ed640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5efc0087d0 0x7f5efc00d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-04T21:01:16.008 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:00:07.524 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:01:16.032 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:00:07.549 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:01:17.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 10 monclient(hunting): tick 2024-04-04T21:01:17.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 10 monclient(hunting): _check_auth_tickets 2024-04-04T21:01:17.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 1 monclient(hunting): continuing hunt 2024-04-04T21:01:17.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-04T21:01:17.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5efc00aa90 msgr2=0x7f5efc005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-04T21:01:17.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5efc00aa90 0x7f5efc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:17.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f5efc0087d0 msgr2=0x7f5efc00d0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-04T21:01:17.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5efc0087d0 0x7f5efc00d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:17.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f5efc009f00 legacy=0x7f5efc00a2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-04T21:01:17.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 10 monclient: _add_conns ranks=[3,1] 2024-04-04T21:01:17.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 0x7f5efc00dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:17.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 10 monclient(hunting): picked mon.noname-a con 0x7f5efc00d6d0 addr v2:172.21.15.106:3300/0 2024-04-04T21:01:17.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5efc00e1b0 addr v1:172.21.15.203:6789/0 2024-04-04T21:01:17.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f180b8c90 con 0x7f5efc00e1b0 2024-04-04T21:01:17.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:17.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f180a9f40 con 0x7f5efc00d6d0 2024-04-04T21:01:17.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.520+0000 7f5f167fc640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:17.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.597+0000 7f5f17fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f5efc00e1b0 0x7f5efc0127a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:44056/0 (socket says 172.21.15.106:44056) 2024-04-04T21:01:17.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.597+0000 7f5f17fff640 1 -- 172.21.15.106:0/1585801252 learned_addr learned my addr 172.21.15.106:0/1585801252 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:01:17.784 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.784+0000 7f5f1c9ec640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 0x7f5efc00dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:17.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.786+0000 7f5f1c9ec640 10 monclient(hunting): get_auth_request con 0x7f5efc00d6d0 auth_method 0 2024-04-04T21:01:17.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.786+0000 7f5f1c9ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:17.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.786+0000 7f5f1c9ec640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:17.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.786+0000 7f5f1c9ec640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:17.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.815+0000 7f5f1c9ec640 1 -- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 msgr2=0x7f5efc00dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-04T21:01:17.816 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.815+0000 7f5f1c9ec640 1 -- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 msgr2=0x7f5efc00dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-04T21:01:17.816 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.815+0000 7f5f1c9ec640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 0x7f5efc00dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-04T21:01:17.816 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:17.815+0000 7f5f1c9ec640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 0x7f5efc00dae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-04T21:01:18.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.016+0000 7f5f1c9ec640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 0x7f5efc00dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:18.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.016+0000 7f5f1c9ec640 10 monclient(hunting): get_auth_request con 0x7f5efc00d6d0 auth_method 0 2024-04-04T21:01:18.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.016+0000 7f5f1c9ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:18.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.016+0000 7f5f1c9ec640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:18.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.016+0000 7f5f1c9ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-04T21:01:18.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.017+0000 7f5f1c9ec640 1 -- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 msgr2=0x7f5efc00dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-04T21:01:18.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.017+0000 7f5f1c9ec640 1 -- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 msgr2=0x7f5efc00dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-04T21:01:18.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.017+0000 7f5f1c9ec640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 0x7f5efc00dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-04T21:01:18.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.017+0000 7f5f1c9ec640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 0x7f5efc00dae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-04T21:01:18.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.517+0000 7f5f1c9ec640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 0x7f5efc00dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:18.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.517+0000 7f5f1c9ec640 10 monclient(hunting): get_auth_request con 0x7f5efc00d6d0 auth_method 0 2024-04-04T21:01:18.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.517+0000 7f5f1c9ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:18.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.517+0000 7f5f1c9ec640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:18.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.517+0000 7f5f1c9ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-04T21:01:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.518+0000 7f5f1c9ec640 1 -- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 msgr2=0x7f5efc00dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-04T21:01:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.518+0000 7f5f1c9ec640 1 -- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 msgr2=0x7f5efc00dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-04T21:01:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.518+0000 7f5f1c9ec640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 0x7f5efc00dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-04T21:01:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:18.518+0000 7f5f1c9ec640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 0x7f5efc00dae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-04T21:01:19.734 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:19.733+0000 7f5f1c9ec640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 0x7f5efc00dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:19.734 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:19.734+0000 7f5f1c9ec640 10 monclient(hunting): get_auth_request con 0x7f5efc00d6d0 auth_method 0 2024-04-04T21:01:19.735 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:19.734+0000 7f5f1c9ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:19.735 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:19.734+0000 7f5f1c9ec640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:19.735 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:19.734+0000 7f5f1c9ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-04T21:01:19.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:19.736+0000 7f5f1c9ec640 1 -- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 msgr2=0x7f5efc00dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-04T21:01:19.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:19.736+0000 7f5f1c9ec640 1 -- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 msgr2=0x7f5efc00dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-04T21:01:19.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:19.736+0000 7f5f1c9ec640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 0x7f5efc00dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-04T21:01:19.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:19.736+0000 7f5f1c9ec640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 0x7f5efc00dae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-04T21:01:20.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.520+0000 7f5f167fc640 10 monclient(hunting): tick 2024-04-04T21:01:20.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.520+0000 7f5f167fc640 10 monclient(hunting): _check_auth_tickets 2024-04-04T21:01:20.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.520+0000 7f5f167fc640 1 monclient(hunting): continuing hunt 2024-04-04T21:01:20.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.520+0000 7f5f167fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-04T21:01:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.520+0000 7f5f167fc640 1 -- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 msgr2=0x7f5efc00dae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-04T21:01:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.520+0000 7f5f167fc640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.106:3300/0 conn(0x7f5efc00d6d0 0x7f5efc00dae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.521+0000 7f5f167fc640 1 -- 172.21.15.106:0/1585801252 >> v1:172.21.15.203:6789/0 conn(0x7f5efc00e1b0 legacy=0x7f5efc0127a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.521+0000 7f5f1d1ed640 1 -- 172.21.15.106:0/1585801252 reap_dead start 2024-04-04T21:01:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.521+0000 7f5f167fc640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-04-04T21:01:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.521+0000 7f5f167fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5efc0087d0 addr v1:172.21.15.106:6789/0 2024-04-04T21:01:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.521+0000 7f5f167fc640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.203:3300/0 conn(0x7f5efc00db50 0x7f5efc00d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.521+0000 7f5f167fc640 10 monclient(hunting): picked mon.noname-c con 0x7f5efc00db50 addr v2:172.21.15.203:3300/0 2024-04-04T21:01:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.521+0000 7f5f167fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5efc013f40 addr v1:172.21.15.203:6789/0 2024-04-04T21:01:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.521+0000 7f5f167fc640 1 -- 172.21.15.106:0/1585801252 --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f180b2e00 con 0x7f5efc0087d0 2024-04-04T21:01:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.521+0000 7f5f167fc640 1 -- 172.21.15.106:0/1585801252 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc014a80 con 0x7f5efc013f40 2024-04-04T21:01:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.521+0000 7f5f167fc640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.521+0000 7f5f167fc640 1 -- 172.21.15.106:0/1585801252 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f180a9f40 con 0x7f5efc00db50 2024-04-04T21:01:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.521+0000 7f5f167fc640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:20.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.647+0000 7f5f17fff640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.203:3300/0 conn(0x7f5efc00db50 0x7f5efc00d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:20.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.648+0000 7f5f17fff640 10 monclient(hunting): get_auth_request con 0x7f5efc00db50 auth_method 0 2024-04-04T21:01:20.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.648+0000 7f5f17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:20.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.648+0000 7f5f17fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:20.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.648+0000 7f5f17fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:20.658 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.657+0000 7f5f17fff640 1 -- 172.21.15.106:0/1585801252 >> v2:172.21.15.203:3300/0 conn(0x7f5efc00db50 msgr2=0x7f5efc00d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-04T21:01:20.658 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.657+0000 7f5f17fff640 1 -- 172.21.15.106:0/1585801252 >> v2:172.21.15.203:3300/0 conn(0x7f5efc00db50 msgr2=0x7f5efc00d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-04T21:01:20.658 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.657+0000 7f5f17fff640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.203:3300/0 conn(0x7f5efc00db50 0x7f5efc00d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-04T21:01:20.658 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.657+0000 7f5f17fff640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.203:3300/0 conn(0x7f5efc00db50 0x7f5efc00d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-04T21:01:20.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.854+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845090012 0 0) 0x7f5efc014a80 con 0x7f5efc013f40 2024-04-04T21:01:20.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.854+0000 7f5f177fe640 10 monclient(hunting): _negotiate 2024-04-04T21:01:20.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.854+0000 7f5f177fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:20.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.854+0000 7f5f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:20.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.854+0000 7f5f177fe640 10 monclient(hunting): my global_id is 4102 2024-04-04T21:01:20.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.855+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f08003520 con 0x7f5efc013f40 2024-04-04T21:01:20.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.863+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576505657 0 0) 0x7f5f180b2e00 con 0x7f5efc0087d0 2024-04-04T21:01:20.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.863+0000 7f5f177fe640 10 monclient(hunting): _negotiate 2024-04-04T21:01:20.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.863+0000 7f5f177fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:20.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.863+0000 7f5f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:20.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.863+0000 7f5f177fe640 10 monclient(hunting): my global_id is 4098 2024-04-04T21:01:20.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.863+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5efc014a80 con 0x7f5efc0087d0 2024-04-04T21:01:20.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.867+0000 7f5f17fff640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.203:3300/0 conn(0x7f5efc00db50 0x7f5efc00d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:20.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.867+0000 7f5f17fff640 10 monclient(hunting): get_auth_request con 0x7f5efc00db50 auth_method 0 2024-04-04T21:01:20.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.867+0000 7f5f17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:20.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.867+0000 7f5f17fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:20.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.867+0000 7f5f17fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-04T21:01:20.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.869+0000 7f5f17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:20.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.869+0000 7f5f17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:20.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.869+0000 7f5f17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:20.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 <== mon.0 v1:172.21.15.106:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1140848299 0 0) 0x7f5efc014a80 con 0x7f5efc0087d0 2024-04-04T21:01:20.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 >> v2:172.21.15.203:3300/0 conn(0x7f5efc00db50 msgr2=0x7f5efc00d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:01:20.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 1 --2- 172.21.15.106:0/1585801252 >> v2:172.21.15.203:3300/0 conn(0x7f5efc00db50 0x7f5efc00d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:20.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 >> v1:172.21.15.203:6789/0 conn(0x7f5efc013f40 legacy=0x7f5efc014370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:20.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 10 monclient: _finish_hunting 0 2024-04-04T21:01:20.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 1 monclient: found mon.noname-a-legacy 2024-04-04T21:01:20.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:20.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.106:6789/0 2024-04-04T21:01:20.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 --> v1:172.21.15.106:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc014cd0 con 0x7f5efc0087d0 2024-04-04T21:01:20.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 10 monclient: _finish_auth 0 2024-04-04T21:01:20.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 10 monclient: _check_auth_tickets 2024-04-04T21:01:20.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:20.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 <== mon.0 v1:172.21.15.106:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 141466616 0 0) 0x7f5f0c00b2e0 con 0x7f5efc0087d0 2024-04-04T21:01:20.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:20.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-04T21:01:20.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 10 monclient: dump: 2024-04-04T21:01:20.901 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:01:20.901 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:20.902 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:01:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] but i'm connected to v1:172.21.15.106:6789/0, reconnecting 2024-04-04T21:01:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 >> v1:172.21.15.106:6789/0 conn(0x7f5efc0087d0 legacy=0x7f5efc00a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:20.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-04T21:01:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 1 --2- 172.21.15.106:0/1585801252 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f08003b40 0x7f5f08003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 10 monclient(hunting): picked mon.a con 0x7f5f08003b40 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 1 --2- 172.21.15.106:0/1585801252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f08004800 0x7f5f08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 10 monclient(hunting): picked mon.b con 0x7f5f08004800 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 1 --2- 172.21.15.106:0/1585801252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f080051c0 0x7f5f0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.898+0000 7f5f177fe640 10 monclient(hunting): picked mon.c con 0x7f5f080051c0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.899+0000 7f5f177fe640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.899+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f180a9f40 con 0x7f5f08003b40 2024-04-04T21:01:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.899+0000 7f5f177fe640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.899+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0800c090 con 0x7f5f08004800 2024-04-04T21:01:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.899+0000 7f5f177fe640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.899+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0800c290 con 0x7f5f080051c0 2024-04-04T21:01:20.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.899+0000 7f5f177fe640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.899+0000 7f5f177fe640 10 monclient(hunting): _finish_auth 0 2024-04-04T21:01:20.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.927+0000 7f5f17fff640 1 --2- 172.21.15.106:0/1585801252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f08004800 0x7f5f08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:20.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.928+0000 7f5f17fff640 10 monclient(hunting): get_auth_request con 0x7f5f08004800 auth_method 0 2024-04-04T21:01:20.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.928+0000 7f5f17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:20.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.928+0000 7f5f17fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:20.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.928+0000 7f5f17fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-04T21:01:20.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.929+0000 7f5f17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:20.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.929+0000 7f5f17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:20.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.929+0000 7f5f17fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-04T21:01:20.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.943+0000 7f5f1c9ec640 1 --2- 172.21.15.106:0/1585801252 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f08003b40 0x7f5f08003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:20.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f17fff640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-04-04T21:01:20.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f17fff640 1 -- 172.21.15.106:0/1585801252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f080051c0 msgr2=0x7f5f0800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:01:20.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f17fff640 1 --2- 172.21.15.106:0/1585801252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f080051c0 0x7f5f0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:20.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f17fff640 1 -- 172.21.15.106:0/1585801252 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f08003b40 msgr2=0x7f5f08003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:01:20.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f17fff640 1 --2- 172.21.15.106:0/1585801252 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f08003b40 0x7f5f08003f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:20.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f17fff640 10 monclient: _finish_hunting 0 2024-04-04T21:01:20.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f17fff640 1 monclient: found mon.b 2024-04-04T21:01:20.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:20.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f17fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:20.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f17fff640 1 -- 172.21.15.106:0/1585801252 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc014a80 con 0x7f5f08004800 2024-04-04T21:01:20.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f17fff640 10 monclient: _finish_auth 0 2024-04-04T21:01:20.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f17fff640 10 monclient: _check_auth_tickets 2024-04-04T21:01:20.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:20.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f1ec50640 5 monclient: authenticate success, global_id 4098 2024-04-04T21:01:20.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f1ec50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:01:20.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.944+0000 7f5f1d1ed640 1 -- 172.21.15.106:0/1585801252 reap_dead start 2024-04-04T21:01:20.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.951+0000 7f5f17fff640 1 --2- 172.21.15.106:0/1585801252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f08004800 0x7f5f08004c80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5efc008c00 tx=0x7f5f08005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70e638de71a807de server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:20.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.995+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f04036020 con 0x7f5f08004800 2024-04-04T21:01:20.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.995+0000 7f5f177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:20.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.995+0000 7f5f177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:20.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.995+0000 7f5f177fe640 10 monclient: dump: 2024-04-04T21:01:20.997 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:01:20.997 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:20.997 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:20.997 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:20.997 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:20.998 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:20.998 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:20.998 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:20.998 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:20.998 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:01:20.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.995+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f04032500 con 0x7f5f08004800 2024-04-04T21:01:20.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.995+0000 7f5f1ec50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:01:20.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.995+0000 7f5f177fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:20.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.995+0000 7f5f1ec50640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:20.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.995+0000 7f5f1ec50640 4 set_mon_vals no callback set 2024-04-04T21:01:20.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.995+0000 7f5f177fe640 1 -- 172.21.15.106:0/1585801252 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f04004950 con 0x7f5f08004800 2024-04-04T21:01:20.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.995+0000 7f5f177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:20.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.995+0000 7f5f177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:20.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.995+0000 7f5f177fe640 10 monclient: dump: 2024-04-04T21:01:20.999 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:01:20.999 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:21.000 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:21.000 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:21.000 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:21.000 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:21.000 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:21.000 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:21.000 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:21.000 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:01:21.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.996+0000 7f5f1ec50640 10 monclient: shutdown 2024-04-04T21:01:21.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.996+0000 7f5f1ec50640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:21.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.996+0000 7f5f1ec50640 1 -- 172.21.15.106:0/1585801252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f08004800 msgr2=0x7f5f08004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:21.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.996+0000 7f5f1ec50640 1 --2- 172.21.15.106:0/1585801252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f08004800 0x7f5f08004c80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5efc008c00 tx=0x7f5f08005830 comp rx=0 tx=0).stop 2024-04-04T21:01:21.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.996+0000 7f5f1ec50640 1 -- 172.21.15.106:0/1585801252 shutdown_connections 2024-04-04T21:01:21.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.996+0000 7f5f1ec50640 1 --2- 172.21.15.106:0/1585801252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f08004800 0x7f5f08004c80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:21.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.996+0000 7f5f1ec50640 1 -- 172.21.15.106:0/1585801252 >> 172.21.15.106:0/1585801252 conn(0x7f5f180b7ec0 msgr2=0x7f5f180b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:21.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.996+0000 7f5f1ec50640 1 -- 172.21.15.106:0/1585801252 shutdown_connections 2024-04-04T21:01:21.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.996+0000 7f5f1ec50640 1 -- 172.21.15.106:0/1585801252 wait complete. 2024-04-04T21:01:21.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.997+0000 7f5f1ec50640 10 monclient: build_initial_monmap 2024-04-04T21:01:21.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.997+0000 7f5f1ec50640 10 monclient: monmap: 2024-04-04T21:01:21.004 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:01:21.004 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:21.004 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:01:21.004 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:01:21.004 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:21.004 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:21.004 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:21.004 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:21.004 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:21.005 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:01:21.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.997+0000 7f5f1ec50640 1 librados: starting msgr at 2024-04-04T21:01:21.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.997+0000 7f5f1ec50640 1 librados: starting objecter 2024-04-04T21:01:21.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.997+0000 7f5f1ec50640 1 Processor -- start 2024-04-04T21:01:21.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.998+0000 7f5f1ec50640 1 -- start start 2024-04-04T21:01:21.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.998+0000 7f5f1ec50640 1 librados: setting wanted keys 2024-04-04T21:01:21.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.998+0000 7f5f1ec50640 1 librados: calling monclient init 2024-04-04T21:01:21.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.998+0000 7f5f1ec50640 10 monclient: init 2024-04-04T21:01:21.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:21.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:01:21.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f08004800 0x7f5f180b1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:21.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 10 monclient(hunting): picked mon.noname-c con 0x7f5f08004800 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:21.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f181db4b0 0x7f5f18141fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:21.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 10 monclient(hunting): picked mon.noname-b con 0x7f5f181db4b0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:21.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f181da8c0 0x7f5f181d73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:21.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 10 monclient(hunting): picked mon.noname-a con 0x7f5f181da8c0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:21.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:21.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f180b28c0 con 0x7f5f181da8c0 2024-04-04T21:01:21.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:21.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f180b25c0 con 0x7f5f181db4b0 2024-04-04T21:01:21.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:21.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f180b2a40 con 0x7f5f08004800 2024-04-04T21:01:21.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:21.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:20.999+0000 7f5f1ec50640 10 monclient(hunting): authenticate will time out at 1330.266296s 2024-04-04T21:01:21.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.012+0000 7f5f17fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f181db4b0 0x7f5f18141fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:21.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.012+0000 7f5f17fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f181db4b0 0x7f5f18141fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50850/0 (socket says 172.21.15.106:50850) 2024-04-04T21:01:21.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.012+0000 7f5f17fff640 1 -- 172.21.15.106:0/3051821553 learned_addr learned my addr 172.21.15.106:0/3051821553 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:01:21.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.012+0000 7f5f17fff640 10 monclient(hunting): get_auth_request con 0x7f5f181db4b0 auth_method 0 2024-04-04T21:01:21.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.012+0000 7f5f17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:21.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.012+0000 7f5f17fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:21.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.012+0000 7f5f17fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:21.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.014+0000 7f5f17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:21.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.014+0000 7f5f17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:21.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.014+0000 7f5f17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:21.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.021+0000 7f5f1c9ec640 1 --2- 172.21.15.106:0/3051821553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f08004800 0x7f5f180b1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:21.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.023+0000 7f5f17fff640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-04-04T21:01:21.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.023+0000 7f5f17fff640 1 -- 172.21.15.106:0/3051821553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f08004800 msgr2=0x7f5f180b1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:21.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.023+0000 7f5f17fff640 1 --2- 172.21.15.106:0/3051821553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f08004800 0x7f5f180b1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:21.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.023+0000 7f5f17fff640 1 -- 172.21.15.106:0/3051821553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f181da8c0 msgr2=0x7f5f181d73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:21.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.023+0000 7f5f17fff640 1 --2- 172.21.15.106:0/3051821553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f181da8c0 0x7f5f181d73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:21.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.023+0000 7f5f17fff640 10 monclient: _finish_hunting 0 2024-04-04T21:01:21.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.023+0000 7f5f17fff640 1 monclient: found mon.noname-b 2024-04-04T21:01:21.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.023+0000 7f5f17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:21.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.023+0000 7f5f17fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:01:21.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.023+0000 7f5f17fff640 1 -- 172.21.15.106:0/3051821553 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f181daf90 con 0x7f5f181db4b0 2024-04-04T21:01:21.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.025+0000 7f5f17fff640 1 --2- 172.21.15.106:0/3051821553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f181db4b0 0x7f5f18141fc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5f04035040 tx=0x7f5f04003ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:21.026 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.025+0000 7f5f157fa640 10 client.?.objecter ms_handle_connect 0x7f5f181db4b0 2024-04-04T21:01:21.026 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.025+0000 7f5f157fa640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:21.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f157fa640 1 -- 172.21.15.106:0/3051821553 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f04036070 con 0x7f5f181db4b0 2024-04-04T21:01:21.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:21.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f157fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:21.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f157fa640 10 monclient: dump: 2024-04-04T21:01:21.062 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:01:21.062 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:21.062 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:21.062 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:21.062 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:21.062 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:21.063 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:21.063 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:21.063 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:21.063 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:01:21.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f157fa640 10 monclient: _finish_auth 0 2024-04-04T21:01:21.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f157fa640 10 monclient: _check_auth_tickets 2024-04-04T21:01:21.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:21.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f157fa640 1 -- 172.21.15.106:0/3051821553 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0403b830 con 0x7f5f181db4b0 2024-04-04T21:01:21.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f157fa640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:21.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f1ec50640 5 monclient: authenticate success, global_id 4114 2024-04-04T21:01:21.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f157fa640 1 -- 172.21.15.106:0/3051821553 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0402e040 con 0x7f5f181db4b0 2024-04-04T21:01:21.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f1ec50640 10 monclient: _renew_subs 2024-04-04T21:01:21.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f1ec50640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:21.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f1ec50640 1 -- 172.21.15.106:0/3051821553 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f181d7900 con 0x7f5f181db4b0 2024-04-04T21:01:21.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:21.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:21.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f157fa640 10 monclient: dump: 2024-04-04T21:01:21.064 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:01:21.064 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:21.064 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:21.064 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:21.064 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:21.064 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:21.065 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:21.065 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:21.065 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:21.065 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:01:21.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f16ffd640 4 set_mon_vals no callback set 2024-04-04T21:01:21.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f1ec50640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:21.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f1ec50640 10 monclient: _renew_subs 2024-04-04T21:01:21.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f1ec50640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:21.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f1ec50640 1 -- 172.21.15.106:0/3051821553 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f18139180 con 0x7f5f181db4b0 2024-04-04T21:01:21.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.060+0000 7f5f1ec50640 1 librados: init done 2024-04-04T21:01:21.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.061+0000 7f5f1ec50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:21.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.061+0000 7f5f1ec50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:21.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.061+0000 7f5f1ec50640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:21.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.061+0000 7f5f1ec50640 1 -- 172.21.15.106:0/3051821553 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f1804f050 con 0x7f5f181db4b0 2024-04-04T21:01:21.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.245+0000 7f5f157fa640 1 -- 172.21.15.106:0/3051821553 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5f04039020 con 0x7f5f181db4b0 2024-04-04T21:01:21.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.252+0000 7f5f157fa640 1 -- 172.21.15.106:0/3051821553 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5f0404e9c0 con 0x7f5f181db4b0 2024-04-04T21:01:21.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.252+0000 7f5f157fa640 10 client.4114.objecter ms_dispatch 0x7f5f18000b90 osd_map(1..1 src has 1..1) v4 2024-04-04T21:01:21.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.252+0000 7f5f157fa640 3 client.4114.objecter handle_osd_map got epochs [1,1] > 0 2024-04-04T21:01:21.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.252+0000 7f5f157fa640 3 client.4114.objecter handle_osd_map decoding full epoch 1 2024-04-04T21:01:21.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.252+0000 7f5f157fa640 20 client.4114.objecter dump_active .. 0 homeless 2024-04-04T21:01:21.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.253+0000 7f5f157fa640 1 -- 172.21.15.106:0/3051821553 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5f04004950 con 0x7f5f181db4b0 2024-04-04T21:01:21.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.253+0000 7f5f157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:21.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.253+0000 7f5f157fa640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:21.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.320+0000 7f5f1ec50640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-04T21:01:21.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.320+0000 7f5f1ec50640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-04T21:01:21.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.320+0000 7f5f1ec50640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:21.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.320+0000 7f5f1ec50640 1 -- 172.21.15.106:0/3051821553 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5f18067d40 con 0x7f5f181db4b0 2024-04-04T21:01:21.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.631+0000 7f5f157fa640 1 -- 172.21.15.106:0/3051821553 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5f0404e480 con 0x7f5f181db4b0 2024-04-04T21:01:21.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.631+0000 7f5f157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-04T21:01:21.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.631+0000 7f5f157fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-04-04T21:01:21.632 INFO:teuthology.orchestra.run.smithi106.stderr:adjusted tunables profile to default 2024-04-04T21:01:21.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 10 librados: watch_flush enter 2024-04-04T21:01:21.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 10 librados: watch_flush exit 2024-04-04T21:01:21.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 20 client.4114.objecter shutdown clearing up homeless session... 2024-04-04T21:01:21.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 10 client.4114.objecter successfully canceled tick 2024-04-04T21:01:21.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 10 monclient: shutdown 2024-04-04T21:01:21.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:21.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 1 -- 172.21.15.106:0/3051821553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f181db4b0 msgr2=0x7f5f18141fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:21.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 1 --2- 172.21.15.106:0/3051821553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f181db4b0 0x7f5f18141fc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5f04035040 tx=0x7f5f04003ce0 comp rx=0 tx=0).stop 2024-04-04T21:01:21.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 1 -- 172.21.15.106:0/3051821553 shutdown_connections 2024-04-04T21:01:21.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 1 --2- 172.21.15.106:0/3051821553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f181da8c0 0x7f5f181d73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:21.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 1 --2- 172.21.15.106:0/3051821553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f181db4b0 0x7f5f18141fc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:21.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 1 --2- 172.21.15.106:0/3051821553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f08004800 0x7f5f180b1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:21.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 1 -- 172.21.15.106:0/3051821553 >> 172.21.15.106:0/3051821553 conn(0x7f5f180b7ec0 msgr2=0x7f5f18074580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:21.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 1 -- 172.21.15.106:0/3051821553 shutdown_connections 2024-04-04T21:01:21.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 1 -- 172.21.15.106:0/3051821553 wait complete. 2024-04-04T21:01:21.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:21.632+0000 7f5f1ec50640 1 librados: shutdown 2024-04-04T21:01:21.645 INFO:tasks.ceph:check_enable_crimson: False 2024-04-04T21:01:21.645 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-04T21:01:21.645 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:01:21.645 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-04T21:01:21.647 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:21.645+0000 7f6dbac4a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-04T21:01:21.675 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:01:21.675 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-04T21:01:21.691 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:21.691+0000 7f05a03b8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-04T21:01:21.705 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:21.704+0000 7f6dbac4a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-04T21:01:21.733 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:01:21.733 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-04T21:01:21.752 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:21.752+0000 7f05a03b8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-04T21:01:21.801 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:01:21.801 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-04T21:01:21.869 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:01:21.869 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-04T21:01:21.896 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:01:21.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-04T21:01:21.962 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:01:21.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-04T21:01:22.030 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-04T21:01:22.030 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-04T21:01:22.097 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 28337134-5253-454a-bcda-47267b05b2c7 0 2024-04-04T21:01:22.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.243+0000 7f73988d3640 10 monclient: get_monmap_and_config 2024-04-04T21:01:22.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.243+0000 7f73988d3640 10 monclient: build_initial_monmap 2024-04-04T21:01:22.244 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.243+0000 7f73988d3640 10 monclient: monmap: 2024-04-04T21:01:22.244 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:22.244 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:22.244 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:01:22.243834+0000 2024-04-04T21:01:22.244 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:01:22.243834+0000 2024-04-04T21:01:22.244 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:22.244 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:22.244 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:22.244 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:22.244 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:22.244 INFO:teuthology.orchestra.run.smithi203.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:22.244 INFO:teuthology.orchestra.run.smithi203.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:22.244 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:22.244 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.243+0000 7f73988d3640 1 Processor -- start 2024-04-04T21:01:22.245 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.243+0000 7f73988d3640 1 -- start start 2024-04-04T21:01:22.245 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.243+0000 7f73988d3640 10 monclient: init 2024-04-04T21:01:22.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.244+0000 7f73988d3640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:22.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.244+0000 7f73988d3640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-04-04T21:01:22.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.244+0000 7f73988d3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7394070e50 addr v1:172.21.15.106:6789/0 2024-04-04T21:01:22.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.244+0000 7f73988d3640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7394073110 0x7f7394071230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:22.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.244+0000 7f73988d3640 10 monclient(hunting): picked mon.noname-c con 0x7f7394073110 addr v2:172.21.15.203:3300/0 2024-04-04T21:01:22.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.244+0000 7f73988d3640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7394143eb0 0x7f73941442e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:22.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.244+0000 7f73988d3640 10 monclient(hunting): picked mon.noname-b con 0x7f7394143eb0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.244+0000 7f73988d3640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7394144820 con 0x7f7394070e50 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.244+0000 7f73988d3640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.244+0000 7f73988d3640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7394143a20 con 0x7f7394073110 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.244+0000 7f73988d3640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.244+0000 7f73988d3640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f739413df50 con 0x7f7394143eb0 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.244+0000 7f73988d3640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.244+0000 7f73988d3640 10 monclient(hunting): authenticate will time out at 1322.394467s 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.245+0000 7f7391d9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7394073110 0x7f7394071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.245+0000 7f739259c640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f7394070e50 0x7f739413d840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.203:57278/0 (socket says 172.21.15.203:57278) 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.245+0000 7f739259c640 1 -- 172.21.15.203:0/998446363 learned_addr learned my addr 172.21.15.203:0/998446363 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.245+0000 7f7392d9d640 1 --2- 172.21.15.203:0/998446363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7394143eb0 0x7f73941442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.245+0000 7f7391d9b640 10 monclient(hunting): get_auth_request con 0x7f7394073110 auth_method 0 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.245+0000 7f7391d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.245+0000 7f7391d9b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:22.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.245+0000 7f7391d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:22.249 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.245+0000 7f7392d9d640 10 monclient(hunting): get_auth_request con 0x7f7394143eb0 auth_method 0 2024-04-04T21:01:22.249 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.245+0000 7f7392d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:22.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.245+0000 7f7392d9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:22.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.245+0000 7f7392d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:22.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.246+0000 7f7391d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:22.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.246+0000 7f7391d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:22.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.246+0000 7f7391d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:22.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.247+0000 7f7392d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:22.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.247+0000 7f7392d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:22.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.247+0000 7f7392d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:22.251 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.247+0000 7f7391d9b640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2024-04-04T21:01:22.251 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.247+0000 7f7391d9b640 1 -- 172.21.15.203:0/998446363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7394143eb0 msgr2=0x7f73941442e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:01:22.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.247+0000 7f7391d9b640 1 --2- 172.21.15.203:0/998446363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7394143eb0 0x7f73941442e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.247+0000 7f7391d9b640 1 -- 172.21.15.203:0/998446363 >> v1:172.21.15.106:6789/0 conn(0x7f7394070e50 legacy=0x7f739413d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:22.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.247+0000 7f7391d9b640 10 monclient: _finish_hunting 0 2024-04-04T21:01:22.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.247+0000 7f7391d9b640 1 monclient: found mon.noname-c 2024-04-04T21:01:22.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.247+0000 7f7391d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:22.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.247+0000 7f7391d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:01:22.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.247+0000 7f7391d9b640 1 -- 172.21.15.203:0/998446363 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73940b0dd0 con 0x7f7394073110 2024-04-04T21:01:22.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.249+0000 7f7391d9b640 1 --2- 172.21.15.203:0/998446363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7394073110 0x7f7394071230 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f73840097a0 tx=0x7f738402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb24e381d9190cea server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:22.253 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f739159a640 1 -- 172.21.15.203:0/998446363 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7384038070 con 0x7f7394073110 2024-04-04T21:01:22.253 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f739159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:22.253 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f739159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:01:22.254 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f739159a640 10 monclient: dump: 2024-04-04T21:01:22.254 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:22.254 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:22.254 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.254 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.254 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:22.254 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:22.254 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:22.254 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:22.254 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:22.254 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:22.254 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f739159a640 10 monclient: _finish_auth 0 2024-04-04T21:01:22.254 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f739159a640 10 monclient: _check_auth_tickets 2024-04-04T21:01:22.254 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f739159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f739159a640 1 -- 172.21.15.203:0/998446363 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7384031700 con 0x7f7394073110 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f73988d3640 5 monclient: authenticate success, global_id 4120 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f73988d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f739159a640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f73988d3640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f73988d3640 4 set_mon_vals no callback set 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f739159a640 1 -- 172.21.15.203:0/998446363 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7384031ad0 con 0x7f7394073110 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f739159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f739159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.253+0000 7f739159a640 10 monclient: dump: 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:22.255 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:22.256 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:22.256 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:22.256 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:22.256 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:22.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.254+0000 7f73988d3640 10 monclient: shutdown 2024-04-04T21:01:22.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.254+0000 7f73988d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:22.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.254+0000 7f73988d3640 1 -- 172.21.15.203:0/998446363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7394073110 msgr2=0x7f7394071230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:22.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.254+0000 7f73988d3640 1 --2- 172.21.15.203:0/998446363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7394073110 0x7f7394071230 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f73840097a0 tx=0x7f738402b7a0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.254+0000 7f73988d3640 1 -- 172.21.15.203:0/998446363 shutdown_connections 2024-04-04T21:01:22.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.254+0000 7f73988d3640 1 --2- 172.21.15.203:0/998446363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7394143eb0 0x7f73941442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.254+0000 7f73988d3640 1 --2- 172.21.15.203:0/998446363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7394073110 0x7f7394071230 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.254+0000 7f73988d3640 1 -- 172.21.15.203:0/998446363 >> 172.21.15.203:0/998446363 conn(0x7f73940b62c0 msgr2=0x7f73940b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:22.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.254+0000 7f73988d3640 1 -- 172.21.15.203:0/998446363 shutdown_connections 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.254+0000 7f73988d3640 1 -- 172.21.15.203:0/998446363 wait complete. 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.255+0000 7f73988d3640 10 monclient: build_initial_monmap 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.255+0000 7f73988d3640 10 monclient: monmap: 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.255+0000 7f73988d3640 1 librados: starting msgr at 2024-04-04T21:01:22.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.255+0000 7f73988d3640 1 librados: starting objecter 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.255+0000 7f73988d3640 1 Processor -- start 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 1 -- start start 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 1 librados: setting wanted keys 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 1 librados: calling monclient init 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 10 monclient: init 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7394070e50 0x7f739406be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 10 monclient(hunting): picked mon.noname-b con 0x7f7394070e50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7394143eb0 0x7f73940ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 10 monclient(hunting): picked mon.noname-a con 0x7f7394143eb0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73940add40 0x7f73940b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 10 monclient(hunting): picked mon.noname-c con 0x7f73940add40 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:22.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739414ac60 con 0x7f7394143eb0 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73940723f0 con 0x7f7394070e50 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f739414ade0 con 0x7f73940add40 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.256+0000 7f73988d3640 10 monclient(hunting): authenticate will time out at 1322.406480s 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f739259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7394070e50 0x7f739406be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f7391d9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7394143eb0 0x7f73940ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f739259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7394070e50 0x7f739406be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:47052/0 (socket says 172.21.15.203:47052) 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f739259c640 1 -- 172.21.15.203:0/4265342583 learned_addr learned my addr 172.21.15.203:0/4265342583 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f739259c640 10 monclient(hunting): get_auth_request con 0x7f7394070e50 auth_method 0 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f739259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f739259c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f739259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:22.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f7391d9b640 10 monclient(hunting): get_auth_request con 0x7f7394143eb0 auth_method 0 2024-04-04T21:01:22.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f7391d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:22.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f7391d9b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:22.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f7391d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:22.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f739259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:22.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f739259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:22.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.257+0000 7f739259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:22.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.258+0000 7f7391d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:22.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.258+0000 7f7391d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:22.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.258+0000 7f7391d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:22.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.260+0000 7f739259c640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-04-04T21:01:22.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.260+0000 7f739259c640 1 -- 172.21.15.203:0/4265342583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73940add40 msgr2=0x7f73940b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:22.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.260+0000 7f739259c640 1 --2- 172.21.15.203:0/4265342583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73940add40 0x7f73940b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.260+0000 7f739259c640 1 -- 172.21.15.203:0/4265342583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7394143eb0 msgr2=0x7f73940ae6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:22.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.260+0000 7f739259c640 1 --2- 172.21.15.203:0/4265342583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7394143eb0 0x7f73940ae6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.260+0000 7f739259c640 10 monclient: _finish_hunting 0 2024-04-04T21:01:22.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.260+0000 7f739259c640 1 monclient: found mon.noname-b 2024-04-04T21:01:22.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.260+0000 7f739259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:22.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.260+0000 7f739259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:01:22.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.260+0000 7f739259c640 1 -- 172.21.15.203:0/4265342583 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739406c360 con 0x7f7394070e50 2024-04-04T21:01:22.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.262+0000 7f739259c640 1 --2- 172.21.15.203:0/4265342583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7394070e50 0x7f739406be20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f738800c430 tx=0x7f738800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:22.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.262+0000 7f7382ffd640 10 client.?.objecter ms_handle_connect 0x7f7394070e50 2024-04-04T21:01:22.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.262+0000 7f7382ffd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:22.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f7382ffd640 1 -- 172.21.15.203:0/4265342583 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7388017070 con 0x7f7394070e50 2024-04-04T21:01:22.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f7382ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:22.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f7382ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:22.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f7382ffd640 10 monclient: dump: 2024-04-04T21:01:22.267 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:22.267 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:22.267 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.267 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.267 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:22.267 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:22.267 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:22.267 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:22.267 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:22.267 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f7382ffd640 10 monclient: _finish_auth 0 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f7382ffd640 10 monclient: _check_auth_tickets 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f7382ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f7382ffd640 1 -- 172.21.15.203:0/4265342583 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7388005ce0 con 0x7f7394070e50 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f7382ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f7382ffd640 1 -- 172.21.15.203:0/4265342583 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73880072f0 con 0x7f7394070e50 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f73988d3640 5 monclient: authenticate success, global_id 4123 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f7382ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f7382ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f7382ffd640 10 monclient: dump: 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.268 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:22.269 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:22.269 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:22.269 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:22.269 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:22.269 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:22.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f73988d3640 10 monclient: _renew_subs 2024-04-04T21:01:22.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f73988d3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:22.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f73988d3640 1 -- 172.21.15.203:0/4265342583 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73940af7e0 con 0x7f7394070e50 2024-04-04T21:01:22.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f7390d99640 4 set_mon_vals no callback set 2024-04-04T21:01:22.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f73988d3640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:22.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f73988d3640 10 monclient: _renew_subs 2024-04-04T21:01:22.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f73988d3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:22.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f73988d3640 1 -- 172.21.15.203:0/4265342583 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739406dc00 con 0x7f7394070e50 2024-04-04T21:01:22.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.266+0000 7f73988d3640 1 librados: init done 2024-04-04T21:01:22.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.267+0000 7f73988d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:22.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.267+0000 7f73988d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:22.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.267+0000 7f73988d3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:22.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.267+0000 7f73988d3640 1 -- 172.21.15.203:0/4265342583 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73940aec90 con 0x7f7394070e50 2024-04-04T21:01:22.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.269+0000 7f7382ffd640 1 -- 172.21.15.203:0/4265342583 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f738801a070 con 0x7f7394070e50 2024-04-04T21:01:22.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.269+0000 7f7382ffd640 1 -- 172.21.15.203:0/4265342583 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7388012400 con 0x7f7394070e50 2024-04-04T21:01:22.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.269+0000 7f7382ffd640 10 client.4123.objecter ms_dispatch 0x7f7394000b90 osd_map(2..2 src has 1..2) v4 2024-04-04T21:01:22.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.269+0000 7f7382ffd640 3 client.4123.objecter handle_osd_map got epochs [2,2] > 0 2024-04-04T21:01:22.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.269+0000 7f7382ffd640 3 client.4123.objecter handle_osd_map decoding full epoch 2 2024-04-04T21:01:22.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.269+0000 7f7382ffd640 20 client.4123.objecter dump_active .. 0 homeless 2024-04-04T21:01:22.311 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:22.311+0000 7f05a03b8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-04T21:01:22.314 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:22.314+0000 7f6dbac4a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-04T21:01:22.367 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:22.367+0000 7f05a03b8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-04T21:01:22.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.370+0000 7f7382ffd640 1 -- 172.21.15.203:0/4265342583 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7388021020 con 0x7f7394070e50 2024-04-04T21:01:22.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.370+0000 7f7382ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:22.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.370+0000 7f7382ffd640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:22.376 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:22.375+0000 7f6dbac4a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-04T21:01:22.413 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:22.413+0000 7f05a03b8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-04T21:01:22.420 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:22.420+0000 7f6dbac4a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-04T21:01:22.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.440+0000 7f73988d3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "28337134-5253-454a-bcda-47267b05b2c7", "id": 0}] 2024-04-04T21:01:22.441 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.440+0000 7f73988d3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "28337134-5253-454a-bcda-47267b05b2c7", "id": 0}] 2024-04-04T21:01:22.441 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.440+0000 7f73988d3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:22.441 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.440+0000 7f73988d3640 1 -- 172.21.15.203:0/4265342583 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "28337134-5253-454a-bcda-47267b05b2c7", "id": 0} v 0) v1 -- 0x7f73940008d0 con 0x7f7394070e50 2024-04-04T21:01:22.469 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:22.469+0000 7f05a03b8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-04T21:01:22.475 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:22.475+0000 7f6dbac4a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-04T21:01:22.512 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:22.512+0000 7f05a03b8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-04T21:01:22.519 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:22.519+0000 7f6dbac4a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-04T21:01:22.598 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.598+0000 7f7382ffd640 1 -- 172.21.15.203:0/4265342583 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "28337134-5253-454a-bcda-47267b05b2c7", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7394144820 con 0x7f7394070e50 2024-04-04T21:01:22.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.598+0000 7f7382ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "28337134-5253-454a-bcda-47267b05b2c7", "id": 0}] 2024-04-04T21:01:22.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.598+0000 7f7382ffd640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:01:22.696 INFO:teuthology.orchestra.run.smithi203.stdout:0 2024-04-04T21:01:22.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 10 librados: watch_flush enter 2024-04-04T21:01:22.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 10 librados: watch_flush exit 2024-04-04T21:01:22.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 20 client.4123.objecter shutdown clearing up homeless session... 2024-04-04T21:01:22.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 10 client.4123.objecter successfully canceled tick 2024-04-04T21:01:22.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 10 monclient: shutdown 2024-04-04T21:01:22.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:22.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 1 -- 172.21.15.203:0/4265342583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7394070e50 msgr2=0x7f739406be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:22.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 1 --2- 172.21.15.203:0/4265342583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7394070e50 0x7f739406be20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f738800c430 tx=0x7f738800c8f0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 1 -- 172.21.15.203:0/4265342583 shutdown_connections 2024-04-04T21:01:22.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 1 --2- 172.21.15.203:0/4265342583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73940add40 0x7f73940b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 1 --2- 172.21.15.203:0/4265342583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7394143eb0 0x7f73940ae6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 1 --2- 172.21.15.203:0/4265342583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7394070e50 0x7f739406be20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 1 -- 172.21.15.203:0/4265342583 >> 172.21.15.203:0/4265342583 conn(0x7f73940b62c0 msgr2=0x7f73940b1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:22.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 1 -- 172.21.15.203:0/4265342583 shutdown_connections 2024-04-04T21:01:22.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 1 -- 172.21.15.203:0/4265342583 wait complete. 2024-04-04T21:01:22.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.599+0000 7f73988d3640 1 librados: shutdown 2024-04-04T21:01:22.699 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new ae336a0e-2a90-4323-9f0f-1701fffe6b66 1 2024-04-04T21:01:22.699 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:22.639+0000 7f05a03b8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-04T21:01:22.700 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:22.644+0000 7f6dbac4a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-04T21:01:22.710 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:22.710+0000 7f05a03b8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-04T21:01:22.714 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:22.714+0000 7f6dbac4a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-04T21:01:22.762 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:22.762+0000 7f05a03b8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-04T21:01:22.768 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:22.768+0000 7f6dbac4a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-04T21:01:22.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.800+0000 7f66b43f1640 10 monclient: get_monmap_and_config 2024-04-04T21:01:22.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.800+0000 7f66b43f1640 10 monclient: build_initial_monmap 2024-04-04T21:01:22.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.800+0000 7f66b43f1640 10 monclient: monmap: 2024-04-04T21:01:22.801 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:22.801 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:22.801 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:01:22.801257+0000 2024-04-04T21:01:22.801 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:01:22.801257+0000 2024-04-04T21:01:22.801 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:22.801 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:22.801 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:22.801 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:22.801 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:22.801 INFO:teuthology.orchestra.run.smithi203.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:22.802 INFO:teuthology.orchestra.run.smithi203.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:22.802 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:22.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 1 Processor -- start 2024-04-04T21:01:22.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 1 -- start start 2024-04-04T21:01:22.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 10 monclient: init 2024-04-04T21:01:22.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:22.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-04-04T21:01:22.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66ac0a9a50 addr v1:172.21.15.106:6789/0 2024-04-04T21:01:22.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f66ac0a9e30 0x7f66ac142880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:22.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 10 monclient(hunting): picked mon.noname-a con 0x7f66ac0a9e30 addr v2:172.21.15.106:3300/0 2024-04-04T21:01:22.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f66ac142dc0 0x7f66ac14cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:22.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 10 monclient(hunting): picked mon.noname-c con 0x7f66ac142dc0 addr v2:172.21.15.203:3300/0 2024-04-04T21:01:22.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ac0ab750 con 0x7f66ac0a9a50 2024-04-04T21:01:22.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:22.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ac140120 con 0x7f66ac0a9e30 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ac0aa660 con 0x7f66ac142dc0 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.801+0000 7f66b43f1640 10 monclient(hunting): authenticate will time out at 1322.951911s 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.802+0000 7f66b298e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f66ac142dc0 0x7f66ac14cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.802+0000 7f66b198c640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f66ac0a9e30 0x7f66ac142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.802+0000 7f66b298e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f66ac142dc0 0x7f66ac14cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:44672/0 (socket says 172.21.15.203:44672) 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.802+0000 7f66b298e640 1 -- 172.21.15.203:0/34328131 learned_addr learned my addr 172.21.15.203:0/34328131 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.802+0000 7f66b298e640 10 monclient(hunting): get_auth_request con 0x7f66ac142dc0 auth_method 0 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.802+0000 7f66b298e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.802+0000 7f66b298e640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.802+0000 7f66b298e640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.803+0000 7f66b198c640 10 monclient(hunting): get_auth_request con 0x7f66ac0a9e30 auth_method 0 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.803+0000 7f66b198c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:22.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.803+0000 7f66b198c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:22.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.803+0000 7f66b198c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:22.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.803+0000 7f66b298e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:22.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.803+0000 7f66b298e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:22.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.803+0000 7f66b298e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:22.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.804+0000 7f66b198c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:22.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.804+0000 7f66b198c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:22.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.804+0000 7f66b198c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.805+0000 7f66b298e640 10 monclient(hunting): handle_auth_done global_id 4126 payload 274 2024-04-04T21:01:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.805+0000 7f66b298e640 1 -- 172.21.15.203:0/34328131 >> v2:172.21.15.106:3300/0 conn(0x7f66ac0a9e30 msgr2=0x7f66ac142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:01:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.805+0000 7f66b298e640 1 --2- 172.21.15.203:0/34328131 >> v2:172.21.15.106:3300/0 conn(0x7f66ac0a9e30 0x7f66ac142880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.805+0000 7f66b298e640 1 -- 172.21.15.203:0/34328131 >> v1:172.21.15.106:6789/0 conn(0x7f66ac0a9a50 legacy=0x7f66ac13f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.805+0000 7f66b298e640 10 monclient: _finish_hunting 0 2024-04-04T21:01:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.805+0000 7f66b298e640 1 monclient: found mon.noname-c 2024-04-04T21:01:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.805+0000 7f66b298e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.805+0000 7f66b298e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:01:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.805+0000 7f66b298e640 1 -- 172.21.15.203:0/34328131 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac0aa250 con 0x7f66ac142dc0 2024-04-04T21:01:22.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.807+0000 7f66b298e640 1 --2- 172.21.15.203:0/34328131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66ac142dc0 0x7f66ac14cef0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f66ac0b4550 tx=0x7f66a402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a5c40f753ee26d server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:22.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b118b640 1 -- 172.21.15.203:0/34328131 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a402ca80 con 0x7f66ac142dc0 2024-04-04T21:01:22.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b118b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:22.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b118b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b118b640 10 monclient: dump: 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b118b640 10 monclient: _finish_auth 0 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b118b640 10 monclient: _check_auth_tickets 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b118b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b118b640 1 -- 172.21.15.203:0/34328131 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a402cc20 con 0x7f66ac142dc0 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b118b640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b118b640 1 -- 172.21.15.203:0/34328131 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a40343f0 con 0x7f66ac142dc0 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b118b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b118b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b118b640 10 monclient: dump: 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:22.812 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:22.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b43f1640 5 monclient: authenticate success, global_id 4126 2024-04-04T21:01:22.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b43f1640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:22.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.810+0000 7f66b43f1640 4 set_mon_vals no callback set 2024-04-04T21:01:22.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.811+0000 7f66b43f1640 10 monclient: shutdown 2024-04-04T21:01:22.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.811+0000 7f66b43f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:22.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.811+0000 7f66b43f1640 1 -- 172.21.15.203:0/34328131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66ac142dc0 msgr2=0x7f66ac14cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:22.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.811+0000 7f66b43f1640 1 --2- 172.21.15.203:0/34328131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66ac142dc0 0x7f66ac14cef0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f66ac0b4550 tx=0x7f66a402bee0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.811+0000 7f66b43f1640 1 -- 172.21.15.203:0/34328131 shutdown_connections 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.811+0000 7f66b43f1640 1 --2- 172.21.15.203:0/34328131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66ac142dc0 0x7f66ac14cef0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.811+0000 7f66b43f1640 1 --2- 172.21.15.203:0/34328131 >> v2:172.21.15.106:3300/0 conn(0x7f66ac0a9e30 0x7f66ac142880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.811+0000 7f66b43f1640 1 -- 172.21.15.203:0/34328131 >> 172.21.15.203:0/34328131 conn(0x7f66ac0b8030 msgr2=0x7f66ac0b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.811+0000 7f66b43f1640 1 -- 172.21.15.203:0/34328131 shutdown_connections 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.811+0000 7f66b43f1640 1 -- 172.21.15.203:0/34328131 wait complete. 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.812+0000 7f66b43f1640 10 monclient: build_initial_monmap 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.812+0000 7f66b43f1640 10 monclient: monmap: 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:22.817 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:22.818 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:22.818 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:22.818 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:22.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.813+0000 7f66b43f1640 1 librados: starting msgr at 2024-04-04T21:01:22.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.813+0000 7f66b43f1640 1 librados: starting objecter 2024-04-04T21:01:22.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.813+0000 7f66b43f1640 1 Processor -- start 2024-04-04T21:01:22.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.813+0000 7f66b43f1640 1 -- start start 2024-04-04T21:01:22.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.813+0000 7f66b43f1640 1 librados: setting wanted keys 2024-04-04T21:01:22.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.813+0000 7f66b43f1640 1 librados: calling monclient init 2024-04-04T21:01:22.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.813+0000 7f66b43f1640 10 monclient: init 2024-04-04T21:01:22.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:22.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:01:22.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66ac0a9a50 0x7f66ac144710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:22.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 10 monclient(hunting): picked mon.noname-c con 0x7f66ac0a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66ac0a9e30 0x7f66ac144c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 10 monclient(hunting): picked mon.noname-b con 0x7f66ac0a9e30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66ac142dc0 0x7f66ac1d7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 10 monclient(hunting): picked mon.noname-a con 0x7f66ac142dc0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac1433b0 con 0x7f66ac142dc0 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac143230 con 0x7f66ac0a9e30 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac14d430 con 0x7f66ac0a9a50 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b43f1640 10 monclient(hunting): authenticate will time out at 1322.964311s 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.814+0000 7f66b198c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66ac0a9e30 0x7f66ac144c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b298e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66ac142dc0 0x7f66ac1d7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:22.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b218d640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66ac0a9a50 0x7f66ac144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b198c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66ac0a9e30 0x7f66ac144c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:44676/0 (socket says 172.21.15.203:44676) 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b198c640 1 -- 172.21.15.203:0/1565521993 learned_addr learned my addr 172.21.15.203:0/1565521993 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b198c640 10 monclient(hunting): get_auth_request con 0x7f66ac0a9e30 auth_method 0 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b198c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b198c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b198c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b218d640 10 monclient(hunting): get_auth_request con 0x7f66ac0a9a50 auth_method 0 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b218d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b218d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b218d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b298e640 10 monclient(hunting): get_auth_request con 0x7f66ac142dc0 auth_method 0 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b298e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b298e640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b298e640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b198c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:22.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b198c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:22.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b198c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:22.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b218d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:22.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b218d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:22.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b218d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:22.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b298e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:22.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b298e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:22.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.815+0000 7f66b298e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:22.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.818+0000 7f66b198c640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-04-04T21:01:22.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.818+0000 7f66b198c640 1 -- 172.21.15.203:0/1565521993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66ac0a9a50 msgr2=0x7f66ac144710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:22.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.818+0000 7f66b198c640 1 --2- 172.21.15.203:0/1565521993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66ac0a9a50 0x7f66ac144710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.818+0000 7f66b198c640 1 -- 172.21.15.203:0/1565521993 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66ac142dc0 msgr2=0x7f66ac1d7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:22.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.818+0000 7f66b198c640 1 --2- 172.21.15.203:0/1565521993 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66ac142dc0 0x7f66ac1d7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:22.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.818+0000 7f66b198c640 10 monclient: _finish_hunting 0 2024-04-04T21:01:22.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.818+0000 7f66b198c640 1 monclient: found mon.noname-b 2024-04-04T21:01:22.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.818+0000 7f66b198c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:22.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.818+0000 7f66b198c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:01:22.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.818+0000 7f66b198c640 1 -- 172.21.15.203:0/1565521993 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac1460d0 con 0x7f66ac0a9e30 2024-04-04T21:01:22.823 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:22.819+0000 7f05a03b8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-04T21:01:22.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.820+0000 7f66b198c640 1 --2- 172.21.15.203:0/1565521993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66ac0a9e30 0x7f66ac144c50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f669c00e9b0 tx=0x7f669c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:22.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.820+0000 7f66a2ffd640 10 client.?.objecter ms_handle_connect 0x7f66ac0a9e30 2024-04-04T21:01:22.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.820+0000 7f66a2ffd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:22.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66a2ffd640 1 -- 172.21.15.203:0/1565521993 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c017070 con 0x7f66ac0a9e30 2024-04-04T21:01:22.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:22.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:22.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66a2ffd640 10 monclient: dump: 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66a2ffd640 10 monclient: _finish_auth 0 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66a2ffd640 10 monclient: _check_auth_tickets 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66a2ffd640 1 -- 172.21.15.203:0/1565521993 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669c00c9a0 con 0x7f66ac0a9e30 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:22.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66a2ffd640 1 -- 172.21.15.203:0/1565521993 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c00cd90 con 0x7f66ac0a9e30 2024-04-04T21:01:22.826 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:22.826 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:22.826 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66a2ffd640 10 monclient: dump: 2024-04-04T21:01:22.826 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:22.826 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:22.826 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.826 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:22.826 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:22.826 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:22.826 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:22.826 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:22.826 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:22.826 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:22.826 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66b43f1640 5 monclient: authenticate success, global_id 4129 2024-04-04T21:01:22.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66b098a640 4 set_mon_vals no callback set 2024-04-04T21:01:22.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66b43f1640 10 monclient: _renew_subs 2024-04-04T21:01:22.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66b43f1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:22.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66b43f1640 1 -- 172.21.15.203:0/1565521993 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ac136840 con 0x7f66ac0a9e30 2024-04-04T21:01:22.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66b43f1640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:22.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66b43f1640 10 monclient: _renew_subs 2024-04-04T21:01:22.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66b43f1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:22.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66b43f1640 1 -- 172.21.15.203:0/1565521993 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ac1d8220 con 0x7f66ac0a9e30 2024-04-04T21:01:22.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.824+0000 7f66b43f1640 1 librados: init done 2024-04-04T21:01:22.828 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:22.826+0000 7f6dbac4a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-04T21:01:22.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.825+0000 7f66b43f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:22.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.825+0000 7f66b43f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:22.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.825+0000 7f66b43f1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:22.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.825+0000 7f66b43f1640 1 -- 172.21.15.203:0/1565521993 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ac04f050 con 0x7f66ac0a9e30 2024-04-04T21:01:22.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.828+0000 7f66a2ffd640 1 -- 172.21.15.203:0/1565521993 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f669c01a070 con 0x7f66ac0a9e30 2024-04-04T21:01:22.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.828+0000 7f66a2ffd640 1 -- 172.21.15.203:0/1565521993 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f669c006660 con 0x7f66ac0a9e30 2024-04-04T21:01:22.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.828+0000 7f66a2ffd640 10 client.4129.objecter ms_dispatch 0x7f66ac000b90 osd_map(3..3 src has 1..3) v4 2024-04-04T21:01:22.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.828+0000 7f66a2ffd640 3 client.4129.objecter handle_osd_map got epochs [3,3] > 0 2024-04-04T21:01:22.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.828+0000 7f66a2ffd640 3 client.4129.objecter handle_osd_map decoding full epoch 3 2024-04-04T21:01:22.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.828+0000 7f66a2ffd640 20 client.4129.objecter dump_active .. 0 homeless 2024-04-04T21:01:22.869 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:22.869+0000 7f05a03b8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-04T21:01:22.874 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:22.873+0000 7f6dbac4a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-04T21:01:22.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.928+0000 7f66a2ffd640 1 -- 172.21.15.203:0/1565521993 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f669c005990 con 0x7f66ac0a9e30 2024-04-04T21:01:22.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.928+0000 7f66a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:22.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.928+0000 7f66a2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:23.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.999+0000 7f66b43f1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ae336a0e-2a90-4323-9f0f-1701fffe6b66", "id": 1}] 2024-04-04T21:01:23.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.999+0000 7f66b43f1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ae336a0e-2a90-4323-9f0f-1701fffe6b66", "id": 1}] 2024-04-04T21:01:23.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.999+0000 7f66b43f1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:23.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:22.999+0000 7f66b43f1640 1 -- 172.21.15.203:0/1565521993 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ae336a0e-2a90-4323-9f0f-1701fffe6b66", "id": 1} v 0) v1 -- 0x7f66ac067ed0 con 0x7f66ac0a9e30 2024-04-04T21:01:23.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.100+0000 7f66a2ffd640 1 -- 172.21.15.203:0/1565521993 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ae336a0e-2a90-4323-9f0f-1701fffe6b66", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f669c005c20 con 0x7f66ac0a9e30 2024-04-04T21:01:23.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.100+0000 7f66a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ae336a0e-2a90-4323-9f0f-1701fffe6b66", "id": 1}] 2024-04-04T21:01:23.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.100+0000 7f66a2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:01:23.101 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-04-04T21:01:23.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 10 librados: watch_flush enter 2024-04-04T21:01:23.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 10 librados: watch_flush exit 2024-04-04T21:01:23.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 20 client.4129.objecter shutdown clearing up homeless session... 2024-04-04T21:01:23.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 10 client.4129.objecter successfully canceled tick 2024-04-04T21:01:23.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 10 monclient: shutdown 2024-04-04T21:01:23.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:23.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 1 -- 172.21.15.203:0/1565521993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66ac0a9e30 msgr2=0x7f66ac144c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:23.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 1 --2- 172.21.15.203:0/1565521993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66ac0a9e30 0x7f66ac144c50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f669c00e9b0 tx=0x7f669c00ee70 comp rx=0 tx=0).stop 2024-04-04T21:01:23.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 1 -- 172.21.15.203:0/1565521993 shutdown_connections 2024-04-04T21:01:23.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 1 --2- 172.21.15.203:0/1565521993 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66ac142dc0 0x7f66ac1d7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.103 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 1 --2- 172.21.15.203:0/1565521993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66ac0a9e30 0x7f66ac144c50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.103 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 1 --2- 172.21.15.203:0/1565521993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66ac0a9a50 0x7f66ac144710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.103 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 1 -- 172.21.15.203:0/1565521993 >> 172.21.15.203:0/1565521993 conn(0x7f66ac0b8030 msgr2=0x7f66ac13d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:23.103 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 1 -- 172.21.15.203:0/1565521993 shutdown_connections 2024-04-04T21:01:23.103 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 1 -- 172.21.15.203:0/1565521993 wait complete. 2024-04-04T21:01:23.103 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.101+0000 7f66b43f1640 1 librados: shutdown 2024-04-04T21:01:23.112 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 773087e0-78c9-4d14-bf0a-fb605c21f92f 2 2024-04-04T21:01:23.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.252+0000 7f2c79fb2640 10 monclient: get_monmap_and_config 2024-04-04T21:01:23.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.252+0000 7f2c79fb2640 10 monclient: build_initial_monmap 2024-04-04T21:01:23.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.252+0000 7f2c79fb2640 10 monclient: monmap: 2024-04-04T21:01:23.252 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:23.252 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:23.252 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:01:23.252726+0000 2024-04-04T21:01:23.252 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:01:23.252726+0000 2024-04-04T21:01:23.252 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:23.253 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:23.253 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:23.253 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:23.253 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:23.253 INFO:teuthology.orchestra.run.smithi203.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:23.253 INFO:teuthology.orchestra.run.smithi203.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:23.253 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:23.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.252+0000 7f2c79fb2640 1 Processor -- start 2024-04-04T21:01:23.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.252+0000 7f2c79fb2640 1 -- start start 2024-04-04T21:01:23.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.252+0000 7f2c79fb2640 10 monclient: init 2024-04-04T21:01:23.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.253+0000 7f2c79fb2640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:23.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.253+0000 7f2c79fb2640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-04-04T21:01:23.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.253+0000 7f2c79fb2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2c740a9a50 addr v1:172.21.15.106:6789/0 2024-04-04T21:01:23.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.253+0000 7f2c79fb2640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c740a9e30 0x7f2c74142880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:23.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.253+0000 7f2c79fb2640 10 monclient(hunting): picked mon.noname-b con 0x7f2c740a9e30 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:23.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.253+0000 7f2c79fb2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c74142dc0 addr v1:172.21.15.203:6789/0 2024-04-04T21:01:23.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.253+0000 7f2c79fb2640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c740ab750 con 0x7f2c740a9a50 2024-04-04T21:01:23.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.253+0000 7f2c79fb2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c74140120 con 0x7f2c74142dc0 2024-04-04T21:01:23.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.253+0000 7f2c79fb2640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:23.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.253+0000 7f2c79fb2640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c740aa660 con 0x7f2c740a9e30 2024-04-04T21:01:23.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.253+0000 7f2c79fb2640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:23.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.253+0000 7f2c79fb2640 10 monclient(hunting): authenticate will time out at 1323.403424s 2024-04-04T21:01:23.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.254+0000 7f2c73fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f2c74142dc0 0x7f2c7414cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45500/0 (socket says 172.21.15.203:45500) 2024-04-04T21:01:23.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.254+0000 7f2c73fff640 1 -- 172.21.15.203:0/164853589 learned_addr learned my addr 172.21.15.203:0/164853589 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-04T21:01:23.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.254+0000 7f2c6affd640 1 --2- 172.21.15.203:0/164853589 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c740a9e30 0x7f2c74142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:23.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.254+0000 7f2c6affd640 10 monclient(hunting): get_auth_request con 0x7f2c740a9e30 auth_method 0 2024-04-04T21:01:23.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.254+0000 7f2c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:23.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.254+0000 7f2c6affd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:23.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.254+0000 7f2c6affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:23.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.255+0000 7f2c6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:23.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.255+0000 7f2c6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:23.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.255+0000 7f2c6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:23.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.256+0000 7f2c72ffd640 1 -- 172.21.15.203:0/164853589 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686514178 0 0) 0x7f2c74140120 con 0x7f2c74142dc0 2024-04-04T21:01:23.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.256+0000 7f2c72ffd640 10 monclient(hunting): _negotiate 2024-04-04T21:01:23.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.256+0000 7f2c72ffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:23.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.256+0000 7f2c72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:23.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.256+0000 7f2c72ffd640 10 monclient(hunting): my global_id is 4132 2024-04-04T21:01:23.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.256+0000 7f2c72ffd640 1 -- 172.21.15.203:0/164853589 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c60003520 con 0x7f2c74142dc0 2024-04-04T21:01:23.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.257+0000 7f2c72ffd640 1 -- 172.21.15.203:0/164853589 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755016068 0 0) 0x7f2c740ab750 con 0x7f2c740a9a50 2024-04-04T21:01:23.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.257+0000 7f2c72ffd640 10 monclient(hunting): _negotiate 2024-04-04T21:01:23.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.257+0000 7f2c72ffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:23.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.257+0000 7f2c72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:23.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.257+0000 7f2c72ffd640 10 monclient(hunting): my global_id is 4125 2024-04-04T21:01:23.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.257+0000 7f2c72ffd640 1 -- 172.21.15.203:0/164853589 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c74140120 con 0x7f2c740a9a50 2024-04-04T21:01:23.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.258+0000 7f2c6affd640 10 monclient(hunting): handle_auth_done global_id 4106 payload 274 2024-04-04T21:01:23.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.258+0000 7f2c6affd640 1 -- 172.21.15.203:0/164853589 >> v1:172.21.15.203:6789/0 conn(0x7f2c74142dc0 legacy=0x7f2c7414cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:23.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.258+0000 7f2c6affd640 1 -- 172.21.15.203:0/164853589 >> v1:172.21.15.106:6789/0 conn(0x7f2c740a9a50 legacy=0x7f2c7413f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:23.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.258+0000 7f2c6affd640 10 monclient: _finish_hunting 0 2024-04-04T21:01:23.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.258+0000 7f2c6affd640 1 monclient: found mon.noname-b 2024-04-04T21:01:23.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.258+0000 7f2c6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:23.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.258+0000 7f2c6affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:01:23.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.258+0000 7f2c6affd640 1 -- 172.21.15.203:0/164853589 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c740aa250 con 0x7f2c740a9e30 2024-04-04T21:01:23.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.261+0000 7f2c6affd640 1 --2- 172.21.15.203:0/164853589 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c740a9e30 0x7f2c74142880 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0097d0 tx=0x7f2c5c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab6dbbef34e65d1e server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:23.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c72ffd640 1 -- 172.21.15.203:0/164853589 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c035020 con 0x7f2c740a9e30 2024-04-04T21:01:23.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:23.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c72ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:23.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c72ffd640 10 monclient: dump: 2024-04-04T21:01:23.267 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:23.267 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:23.267 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c72ffd640 10 monclient: _finish_auth 0 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c72ffd640 10 monclient: _check_auth_tickets 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c72ffd640 1 -- 172.21.15.203:0/164853589 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c031800 con 0x7f2c740a9e30 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c72ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c72ffd640 1 -- 172.21.15.203:0/164853589 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c031bd0 con 0x7f2c740a9e30 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:23.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c72ffd640 10 monclient: dump: 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c79fb2640 5 monclient: authenticate success, global_id 4106 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c79fb2640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.266+0000 7f2c79fb2640 4 set_mon_vals no callback set 2024-04-04T21:01:23.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.267+0000 7f2c79fb2640 10 monclient: shutdown 2024-04-04T21:01:23.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.267+0000 7f2c79fb2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:23.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.267+0000 7f2c79fb2640 1 -- 172.21.15.203:0/164853589 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c740a9e30 msgr2=0x7f2c74142880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:23.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.267+0000 7f2c79fb2640 1 --2- 172.21.15.203:0/164853589 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c740a9e30 0x7f2c74142880 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0097d0 tx=0x7f2c5c02b8a0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.267+0000 7f2c79fb2640 1 -- 172.21.15.203:0/164853589 shutdown_connections 2024-04-04T21:01:23.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.267+0000 7f2c79fb2640 1 --2- 172.21.15.203:0/164853589 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c740a9e30 0x7f2c74142880 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.267+0000 7f2c79fb2640 1 -- 172.21.15.203:0/164853589 >> 172.21.15.203:0/164853589 conn(0x7f2c740b8030 msgr2=0x7f2c740b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:23.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.267+0000 7f2c79fb2640 1 -- 172.21.15.203:0/164853589 shutdown_connections 2024-04-04T21:01:23.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.267+0000 7f2c79fb2640 1 -- 172.21.15.203:0/164853589 wait complete. 2024-04-04T21:01:23.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.267+0000 7f2c79fb2640 10 monclient: build_initial_monmap 2024-04-04T21:01:23.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.267+0000 7f2c79fb2640 10 monclient: monmap: 2024-04-04T21:01:23.272 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:23.272 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:23.272 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-04-04T21:01:23.272 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-04-04T21:01:23.272 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:23.272 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:23.272 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:23.272 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:23.272 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:23.272 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:23.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.268+0000 7f2c79fb2640 1 librados: starting msgr at 2024-04-04T21:01:23.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.268+0000 7f2c79fb2640 1 librados: starting objecter 2024-04-04T21:01:23.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.268+0000 7f2c79fb2640 1 Processor -- start 2024-04-04T21:01:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.268+0000 7f2c79fb2640 1 -- start start 2024-04-04T21:01:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.268+0000 7f2c79fb2640 1 librados: setting wanted keys 2024-04-04T21:01:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.268+0000 7f2c79fb2640 1 librados: calling monclient init 2024-04-04T21:01:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.268+0000 7f2c79fb2640 10 monclient: init 2024-04-04T21:01:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.268+0000 7f2c79fb2640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.268+0000 7f2c79fb2640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-04T21:01:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.268+0000 7f2c79fb2640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c740a9a50 0x7f2c740af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.268+0000 7f2c79fb2640 10 monclient(hunting): picked mon.noname-c con 0x7f2c740a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.268+0000 7f2c79fb2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c740a9e30 0x7f2c740afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.268+0000 7f2c79fb2640 10 monclient(hunting): picked mon.noname-a con 0x7f2c740a9e30 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c79fb2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c74142dc0 0x7f2c7413ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c79fb2640 10 monclient(hunting): picked mon.noname-b con 0x7f2c74142dc0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c79fb2640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c79fb2640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c741433d0 con 0x7f2c740a9e30 2024-04-04T21:01:23.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c79fb2640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:23.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c79fb2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c74143250 con 0x7f2c74142dc0 2024-04-04T21:01:23.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c79fb2640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:23.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c79fb2640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7414e610 con 0x7f2c740a9a50 2024-04-04T21:01:23.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c79fb2640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:23.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c79fb2640 10 monclient(hunting): authenticate will time out at 1323.418949s 2024-04-04T21:01:23.274 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f05a03b8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-04T21:01:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c73fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c74142dc0 0x7f2c7413ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c6affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c740a9e30 0x7f2c740afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c737fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c740a9a50 0x7f2c740af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c73fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c74142dc0 0x7f2c7413ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:44678/0 (socket says 172.21.15.203:44678) 2024-04-04T21:01:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c73fff640 1 -- 172.21.15.203:0/877211420 learned_addr learned my addr 172.21.15.203:0/877211420 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-04T21:01:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c73fff640 10 monclient(hunting): get_auth_request con 0x7f2c74142dc0 auth_method 0 2024-04-04T21:01:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c73fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.269+0000 7f2c73fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c6affd640 10 monclient(hunting): get_auth_request con 0x7f2c740a9e30 auth_method 0 2024-04-04T21:01:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c6affd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c6affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c737fe640 10 monclient(hunting): get_auth_request con 0x7f2c740a9a50 auth_method 0 2024-04-04T21:01:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:23.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c737fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:23.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c737fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:23.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:23.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:23.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:23.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:23.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:23.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:23.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:23.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:23.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.270+0000 7f2c737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:23.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.273+0000 7f2c73fff640 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2024-04-04T21:01:23.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.273+0000 7f2c73fff640 1 -- 172.21.15.203:0/877211420 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c740a9a50 msgr2=0x7f2c740af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:23.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.273+0000 7f2c73fff640 1 --2- 172.21.15.203:0/877211420 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c740a9a50 0x7f2c740af950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.273+0000 7f2c73fff640 1 -- 172.21.15.203:0/877211420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c740a9e30 msgr2=0x7f2c740afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:23.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.273+0000 7f2c73fff640 1 --2- 172.21.15.203:0/877211420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c740a9e30 0x7f2c740afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.273+0000 7f2c73fff640 10 monclient: _finish_hunting 0 2024-04-04T21:01:23.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.273+0000 7f2c73fff640 1 monclient: found mon.noname-b 2024-04-04T21:01:23.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.273+0000 7f2c73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:23.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.273+0000 7f2c73fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:01:23.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.273+0000 7f2c73fff640 1 -- 172.21.15.203:0/877211420 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c740b1310 con 0x7f2c74142dc0 2024-04-04T21:01:23.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.275+0000 7f2c73fff640 1 --2- 172.21.15.203:0/877211420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c74142dc0 0x7f2c7413ada0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c00caa0 tx=0x7f2c6c00cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:23.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.275+0000 7f2c70ff9640 10 client.?.objecter ms_handle_connect 0x7f2c74142dc0 2024-04-04T21:01:23.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.275+0000 7f2c70ff9640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:23.281 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c70ff9640 1 -- 172.21.15.203:0/877211420 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c017070 con 0x7f2c74142dc0 2024-04-04T21:01:23.281 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:23.281 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c70ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:23.281 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c70ff9640 10 monclient: dump: 2024-04-04T21:01:23.281 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:23.281 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:23.281 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c70ff9640 10 monclient: _finish_auth 0 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c70ff9640 10 monclient: _check_auth_tickets 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c70ff9640 1 -- 172.21.15.203:0/877211420 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6c00ac20 con 0x7f2c74142dc0 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c79fb2640 5 monclient: authenticate success, global_id 4135 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c79fb2640 10 monclient: _renew_subs 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c79fb2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:23.282 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c79fb2640 1 -- 172.21.15.203:0/877211420 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c740b15f0 con 0x7f2c74142dc0 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c70ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c70ff9640 1 -- 172.21.15.203:0/877211420 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c012640 con 0x7f2c74142dc0 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c70ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c70ff9640 10 monclient: dump: 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:23.283 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:23.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c727fc640 4 set_mon_vals no callback set 2024-04-04T21:01:23.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c79fb2640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:23.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c79fb2640 10 monclient: _renew_subs 2024-04-04T21:01:23.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c79fb2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:23.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c79fb2640 1 -- 172.21.15.203:0/877211420 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7413cf50 con 0x7f2c74142dc0 2024-04-04T21:01:23.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.279+0000 7f2c79fb2640 1 librados: init done 2024-04-04T21:01:23.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.280+0000 7f2c79fb2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:23.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.280+0000 7f2c79fb2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:23.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.280+0000 7f2c79fb2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:23.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.280+0000 7f2c79fb2640 1 -- 172.21.15.203:0/877211420 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7404f050 con 0x7f2c74142dc0 2024-04-04T21:01:23.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.281+0000 7f2c70ff9640 1 -- 172.21.15.203:0/877211420 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2c6c01a070 con 0x7f2c74142dc0 2024-04-04T21:01:23.285 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:23.281+0000 7f6dbac4a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-04T21:01:23.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.387+0000 7f2c70ff9640 1 -- 172.21.15.203:0/877211420 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f2c6c007490 con 0x7f2c74142dc0 2024-04-04T21:01:23.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.387+0000 7f2c70ff9640 10 client.4135.objecter ms_dispatch 0x7f2c74000b90 osd_map(4..4 src has 1..4) v4 2024-04-04T21:01:23.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.387+0000 7f2c70ff9640 3 client.4135.objecter handle_osd_map got epochs [4,4] > 0 2024-04-04T21:01:23.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.387+0000 7f2c70ff9640 3 client.4135.objecter handle_osd_map decoding full epoch 4 2024-04-04T21:01:23.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.387+0000 7f2c70ff9640 20 client.4135.objecter dump_active .. 0 homeless 2024-04-04T21:01:23.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.387+0000 7f2c70ff9640 1 -- 172.21.15.203:0/877211420 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2c6c021020 con 0x7f2c74142dc0 2024-04-04T21:01:23.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.387+0000 7f2c70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:23.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.387+0000 7f2c70ff9640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:23.413 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:23.412+0000 7f05a03b8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-04T21:01:23.415 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:23.414+0000 7f6dbac4a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-04T21:01:23.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.458+0000 7f2c79fb2640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "773087e0-78c9-4d14-bf0a-fb605c21f92f", "id": 2}] 2024-04-04T21:01:23.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.458+0000 7f2c79fb2640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "773087e0-78c9-4d14-bf0a-fb605c21f92f", "id": 2}] 2024-04-04T21:01:23.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.458+0000 7f2c79fb2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:23.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.458+0000 7f2c79fb2640 1 -- 172.21.15.203:0/877211420 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "773087e0-78c9-4d14-bf0a-fb605c21f92f", "id": 2} v 0) v1 -- 0x7f2c740b78c0 con 0x7f2c74142dc0 2024-04-04T21:01:23.538 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:23.538+0000 7f05a03b8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-04T21:01:23.550 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:23.550+0000 7f6dbac4a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-04T21:01:23.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.553+0000 7f2c70ff9640 1 -- 172.21.15.203:0/877211420 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "773087e0-78c9-4d14-bf0a-fb605c21f92f", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2c6c007070 con 0x7f2c74142dc0 2024-04-04T21:01:23.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.553+0000 7f2c70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "773087e0-78c9-4d14-bf0a-fb605c21f92f", "id": 2}] 2024-04-04T21:01:23.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.553+0000 7f2c70ff9640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:01:23.553 INFO:teuthology.orchestra.run.smithi203.stdout:2 2024-04-04T21:01:23.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 10 librados: watch_flush enter 2024-04-04T21:01:23.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 10 librados: watch_flush exit 2024-04-04T21:01:23.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 20 client.4135.objecter shutdown clearing up homeless session... 2024-04-04T21:01:23.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 10 client.4135.objecter successfully canceled tick 2024-04-04T21:01:23.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 10 monclient: shutdown 2024-04-04T21:01:23.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:23.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 1 -- 172.21.15.203:0/877211420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c74142dc0 msgr2=0x7f2c7413ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:23.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 1 --2- 172.21.15.203:0/877211420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c74142dc0 0x7f2c7413ada0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c00caa0 tx=0x7f2c6c00cf60 comp rx=0 tx=0).stop 2024-04-04T21:01:23.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 1 -- 172.21.15.203:0/877211420 shutdown_connections 2024-04-04T21:01:23.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 1 --2- 172.21.15.203:0/877211420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c74142dc0 0x7f2c7413ada0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 1 --2- 172.21.15.203:0/877211420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c740a9e30 0x7f2c740afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 1 --2- 172.21.15.203:0/877211420 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c740a9a50 0x7f2c740af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 1 -- 172.21.15.203:0/877211420 >> 172.21.15.203:0/877211420 conn(0x7f2c740b8030 msgr2=0x7f2c740b8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:23.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 1 -- 172.21.15.203:0/877211420 shutdown_connections 2024-04-04T21:01:23.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 1 -- 172.21.15.203:0/877211420 wait complete. 2024-04-04T21:01:23.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.554+0000 7f2c79fb2640 1 librados: shutdown 2024-04-04T21:01:23.564 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 07da9513-c746-46d1-995f-6f8e91338b84 3 2024-04-04T21:01:23.588 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:23.588+0000 7f05a03b8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-04T21:01:23.596 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:23.596+0000 7f6dbac4a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-04T21:01:23.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.663+0000 7f60b4ac5640 10 monclient: get_monmap_and_config 2024-04-04T21:01:23.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.663+0000 7f60b4ac5640 10 monclient: build_initial_monmap 2024-04-04T21:01:23.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.663+0000 7f60b4ac5640 10 monclient: monmap: 2024-04-04T21:01:23.664 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:23.664 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:23.664 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:01:23.664180+0000 2024-04-04T21:01:23.664 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:01:23.664180+0000 2024-04-04T21:01:23.664 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:23.664 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:23.664 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:23.664 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:23.664 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:23.665 INFO:teuthology.orchestra.run.smithi203.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:23.665 INFO:teuthology.orchestra.run.smithi203.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:23.665 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:23.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 1 Processor -- start 2024-04-04T21:01:23.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 1 -- start start 2024-04-04T21:01:23.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 10 monclient: init 2024-04-04T21:01:23.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:23.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-04-04T21:01:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60b00a9a50 addr v1:172.21.15.203:6789/0 2024-04-04T21:01:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f60b00a9e30 0x7f60b0142880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 10 monclient(hunting): picked mon.noname-c con 0x7f60b00a9e30 addr v2:172.21.15.203:3300/0 2024-04-04T21:01:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60b0142dc0 addr v1:172.21.15.106:6789/0 2024-04-04T21:01:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b00ab750 con 0x7f60b0142dc0 2024-04-04T21:01:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b0140120 con 0x7f60b00a9a50 2024-04-04T21:01:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b00aa660 con 0x7f60b00a9e30 2024-04-04T21:01:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.664+0000 7f60b4ac5640 10 monclient(hunting): authenticate will time out at 1323.814841s 2024-04-04T21:01:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.665+0000 7f60add9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f60b00a9e30 0x7f60b0142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.665+0000 7f60aed9d640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f60b0142dc0 0x7f60b014cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.203:35458/0 (socket says 172.21.15.203:35458) 2024-04-04T21:01:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.665+0000 7f60aed9d640 1 -- 172.21.15.203:0/2984916500 learned_addr learned my addr 172.21.15.203:0/2984916500 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-04T21:01:23.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.666+0000 7f60add9b640 10 monclient(hunting): get_auth_request con 0x7f60b00a9e30 auth_method 0 2024-04-04T21:01:23.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.666+0000 7f60add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:23.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.666+0000 7f60add9b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:23.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.666+0000 7f60add9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:23.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.667+0000 7f60add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:23.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.667+0000 7f60add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:23.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.667+0000 7f60add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:23.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.668+0000 7f60ad59a640 1 -- 172.21.15.203:0/2984916500 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912127980 0 0) 0x7f60b00ab750 con 0x7f60b0142dc0 2024-04-04T21:01:23.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.668+0000 7f60ad59a640 10 monclient(hunting): _negotiate 2024-04-04T21:01:23.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.668+0000 7f60ad59a640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:23.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.668+0000 7f60ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:23.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.668+0000 7f60ad59a640 10 monclient(hunting): my global_id is 4131 2024-04-04T21:01:23.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.668+0000 7f60ad59a640 1 -- 172.21.15.203:0/2984916500 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6098003520 con 0x7f60b0142dc0 2024-04-04T21:01:23.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.671+0000 7f60add9b640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2024-04-04T21:01:23.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.671+0000 7f60add9b640 1 -- 172.21.15.203:0/2984916500 >> v1:172.21.15.203:6789/0 conn(0x7f60b00a9a50 legacy=0x7f60b013f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:23.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.671+0000 7f60add9b640 1 -- 172.21.15.203:0/2984916500 >> v1:172.21.15.106:6789/0 conn(0x7f60b0142dc0 legacy=0x7f60b014cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:23.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.671+0000 7f60add9b640 10 monclient: _finish_hunting 0 2024-04-04T21:01:23.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.671+0000 7f60add9b640 1 monclient: found mon.noname-c 2024-04-04T21:01:23.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.671+0000 7f60add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:23.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.671+0000 7f60add9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:01:23.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.671+0000 7f60add9b640 1 -- 172.21.15.203:0/2984916500 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b00aa250 con 0x7f60b00a9e30 2024-04-04T21:01:23.675 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.675+0000 7f60add9b640 1 --2- 172.21.15.203:0/2984916500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60b00a9e30 0x7f60b0142880 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f60a40097a0 tx=0x7f60a402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f31b9e8d683502f server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:23.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60ad59a640 1 -- 172.21.15.203:0/2984916500 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4035020 con 0x7f60b00a9e30 2024-04-04T21:01:23.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:23.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60ad59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:01:23.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60ad59a640 10 monclient: dump: 2024-04-04T21:01:23.681 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:23.681 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:23.681 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.681 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.681 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:23.681 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:23.681 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:23.681 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:23.681 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:23.682 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:23.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60ad59a640 10 monclient: _finish_auth 0 2024-04-04T21:01:23.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60ad59a640 10 monclient: _check_auth_tickets 2024-04-04T21:01:23.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:23.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60ad59a640 1 -- 172.21.15.203:0/2984916500 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a4031830 con 0x7f60b00a9e30 2024-04-04T21:01:23.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60b4ac5640 5 monclient: authenticate success, global_id 4138 2024-04-04T21:01:23.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60b4ac5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:01:23.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60ad59a640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:23.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60b4ac5640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:23.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60b4ac5640 4 set_mon_vals no callback set 2024-04-04T21:01:23.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60ad59a640 1 -- 172.21.15.203:0/2984916500 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4031c00 con 0x7f60b00a9e30 2024-04-04T21:01:23.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:23.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60ad59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:23.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.679+0000 7f60ad59a640 10 monclient: dump: 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.680+0000 7f60b4ac5640 10 monclient: shutdown 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.680+0000 7f60b4ac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.680+0000 7f60b4ac5640 1 -- 172.21.15.203:0/2984916500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60b00a9e30 msgr2=0x7f60b0142880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.680+0000 7f60b4ac5640 1 --2- 172.21.15.203:0/2984916500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60b00a9e30 0x7f60b0142880 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f60a40097a0 tx=0x7f60a402b8d0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.680+0000 7f60b4ac5640 1 -- 172.21.15.203:0/2984916500 shutdown_connections 2024-04-04T21:01:23.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.680+0000 7f60b4ac5640 1 --2- 172.21.15.203:0/2984916500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60b00a9e30 0x7f60b0142880 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.684 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.680+0000 7f60b4ac5640 1 -- 172.21.15.203:0/2984916500 >> 172.21.15.203:0/2984916500 conn(0x7f60b00b8030 msgr2=0x7f60b00b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:23.684 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.680+0000 7f60b4ac5640 1 -- 172.21.15.203:0/2984916500 shutdown_connections 2024-04-04T21:01:23.684 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.680+0000 7f60b4ac5640 1 -- 172.21.15.203:0/2984916500 wait complete. 2024-04-04T21:01:23.687 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.681+0000 7f60b4ac5640 10 monclient: build_initial_monmap 2024-04-04T21:01:23.687 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.681+0000 7f60b4ac5640 10 monclient: monmap: 2024-04-04T21:01:23.687 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:23.687 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.681+0000 7f60b4ac5640 1 librados: starting msgr at 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.681+0000 7f60b4ac5640 1 librados: starting objecter 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.681+0000 7f60b4ac5640 1 Processor -- start 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 1 -- start start 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 1 librados: setting wanted keys 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 1 librados: calling monclient init 2024-04-04T21:01:23.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 10 monclient: init 2024-04-04T21:01:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:01:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60b00a9a50 0x7f60b01418a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 10 monclient(hunting): picked mon.noname-c con 0x7f60b00a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60b00a9e30 0x7f60b0145990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 10 monclient(hunting): picked mon.noname-b con 0x7f60b00a9e30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60b0142dc0 0x7f60b0145ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 10 monclient(hunting): picked mon.noname-a con 0x7f60b0142dc0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b01433b0 con 0x7f60b0142dc0 2024-04-04T21:01:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b0143230 con 0x7f60b00a9e30 2024-04-04T21:01:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:23.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b014e610 con 0x7f60b00a9a50 2024-04-04T21:01:23.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:23.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.682+0000 7f60b4ac5640 10 monclient(hunting): authenticate will time out at 1323.832486s 2024-04-04T21:01:23.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60add9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60b00a9e30 0x7f60b0145990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:23.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60aed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60b0142dc0 0x7f60b0145ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:23.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60ae59c640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60b00a9a50 0x7f60b01418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:23.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60aed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60b0142dc0 0x7f60b0145ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.203:56504/0 (socket says 172.21.15.203:56504) 2024-04-04T21:01:23.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60aed9d640 1 -- 172.21.15.203:0/1611818766 learned_addr learned my addr 172.21.15.203:0/1611818766 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-04T21:01:23.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60aed9d640 10 monclient(hunting): get_auth_request con 0x7f60b0142dc0 auth_method 0 2024-04-04T21:01:23.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60add9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60b00a9e30 0x7f60b0145990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:44694/0 (socket says 172.21.15.203:44694) 2024-04-04T21:01:23.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:23.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60aed9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:23.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:23.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60add9b640 10 monclient(hunting): get_auth_request con 0x7f60b00a9e30 auth_method 0 2024-04-04T21:01:23.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60add9b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60add9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60ae59c640 10 monclient(hunting): get_auth_request con 0x7f60b00a9a50 auth_method 0 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60ae59c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.683+0000 7f60add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.684+0000 7f60add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.684+0000 7f60add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.684+0000 7f60ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.684+0000 7f60ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.684+0000 7f60ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.684+0000 7f60aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.684+0000 7f60aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.684+0000 7f60aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:23.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.686+0000 7f60add9b640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2024-04-04T21:01:23.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.686+0000 7f60add9b640 1 -- 172.21.15.203:0/1611818766 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60b00a9a50 msgr2=0x7f60b01418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:23.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.686+0000 7f60add9b640 1 --2- 172.21.15.203:0/1611818766 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60b00a9a50 0x7f60b01418a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.686+0000 7f60add9b640 1 -- 172.21.15.203:0/1611818766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60b0142dc0 msgr2=0x7f60b0145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:23.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.686+0000 7f60add9b640 1 --2- 172.21.15.203:0/1611818766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60b0142dc0 0x7f60b0145ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.686+0000 7f60add9b640 10 monclient: _finish_hunting 0 2024-04-04T21:01:23.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.686+0000 7f60add9b640 1 monclient: found mon.noname-b 2024-04-04T21:01:23.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.686+0000 7f60add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:23.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.686+0000 7f60add9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:01:23.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.686+0000 7f60add9b640 1 -- 172.21.15.203:0/1611818766 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b0146570 con 0x7f60b00a9e30 2024-04-04T21:01:23.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.689+0000 7f60add9b640 1 --2- 172.21.15.203:0/1611818766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60b00a9e30 0x7f60b0145990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f60a4033000 tx=0x7f60a40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:23.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.689+0000 7f609effd640 10 client.?.objecter ms_handle_connect 0x7f60b00a9e30 2024-04-04T21:01:23.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.689+0000 7f609effd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:23.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f609effd640 1 -- 172.21.15.203:0/1611818766 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4041070 con 0x7f60b00a9e30 2024-04-04T21:01:23.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f609effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:23.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f609effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:23.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f609effd640 10 monclient: dump: 2024-04-04T21:01:23.694 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:23.694 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:23.694 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.694 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.694 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:23.694 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:23.694 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:23.694 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:23.695 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:23.695 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:23.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f609effd640 10 monclient: _finish_auth 0 2024-04-04T21:01:23.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f609effd640 10 monclient: _check_auth_tickets 2024-04-04T21:01:23.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f609effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:23.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f609effd640 1 -- 172.21.15.203:0/1611818766 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a403b920 con 0x7f60b00a9e30 2024-04-04T21:01:23.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f60b4ac5640 5 monclient: authenticate success, global_id 4144 2024-04-04T21:01:23.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f60b4ac5640 10 monclient: _renew_subs 2024-04-04T21:01:23.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f60b4ac5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:23.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f60b4ac5640 1 -- 172.21.15.203:0/1611818766 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b01341a0 con 0x7f60b00a9e30 2024-04-04T21:01:23.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f609effd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:23.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f609effd640 1 -- 172.21.15.203:0/1611818766 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4035040 con 0x7f60b00a9e30 2024-04-04T21:01:23.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f609effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:23.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f609effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f609effd640 10 monclient: dump: 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f60acd99640 4 set_mon_vals no callback set 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f60b4ac5640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f60b4ac5640 10 monclient: _renew_subs 2024-04-04T21:01:23.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f60b4ac5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:23.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f60b4ac5640 1 -- 172.21.15.203:0/1611818766 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b01346b0 con 0x7f60b00a9e30 2024-04-04T21:01:23.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.692+0000 7f60b4ac5640 1 librados: init done 2024-04-04T21:01:23.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.693+0000 7f60b4ac5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:23.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.693+0000 7f60b4ac5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:23.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.693+0000 7f60b4ac5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:23.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.693+0000 7f60b4ac5640 1 -- 172.21.15.203:0/1611818766 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b004f050 con 0x7f60b00a9e30 2024-04-04T21:01:23.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.696+0000 7f609effd640 1 -- 172.21.15.203:0/1611818766 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f60a404c060 con 0x7f60b00a9e30 2024-04-04T21:01:23.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.696+0000 7f609effd640 1 -- 172.21.15.203:0/1611818766 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f60a403a8d0 con 0x7f60b00a9e30 2024-04-04T21:01:23.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.696+0000 7f609effd640 10 client.4144.objecter ms_dispatch 0x7f60b0000b90 osd_map(5..5 src has 1..5) v4 2024-04-04T21:01:23.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.696+0000 7f609effd640 3 client.4144.objecter handle_osd_map got epochs [5,5] > 0 2024-04-04T21:01:23.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.696+0000 7f609effd640 3 client.4144.objecter handle_osd_map decoding full epoch 5 2024-04-04T21:01:23.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.696+0000 7f609effd640 20 client.4144.objecter dump_active .. 0 homeless 2024-04-04T21:01:23.711 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:23.711+0000 7f05a03b8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-04T21:01:23.713 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:23.713+0000 7f6dbac4a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-04T21:01:23.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.795+0000 7f609effd640 1 -- 172.21.15.203:0/1611818766 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f60a403bdf0 con 0x7f60b00a9e30 2024-04-04T21:01:23.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.795+0000 7f609effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:23.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.795+0000 7f609effd640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:23.811 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:23.811+0000 7f05a03b8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-04T21:01:23.812 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:23.811+0000 7f6dbac4a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-04T21:01:23.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.868+0000 7f60b4ac5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "07da9513-c746-46d1-995f-6f8e91338b84", "id": 3}] 2024-04-04T21:01:23.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.868+0000 7f60b4ac5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "07da9513-c746-46d1-995f-6f8e91338b84", "id": 3}] 2024-04-04T21:01:23.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.869+0000 7f60b4ac5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:23.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.869+0000 7f60b4ac5640 1 -- 172.21.15.203:0/1611818766 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "07da9513-c746-46d1-995f-6f8e91338b84", "id": 3} v 0) v1 -- 0x7f60b0067ed0 con 0x7f60b00a9e30 2024-04-04T21:01:23.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.939+0000 7f609effd640 1 -- 172.21.15.203:0/1611818766 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "07da9513-c746-46d1-995f-6f8e91338b84", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f60a403a510 con 0x7f60b00a9e30 2024-04-04T21:01:23.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.939+0000 7f609effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "07da9513-c746-46d1-995f-6f8e91338b84", "id": 3}] 2024-04-04T21:01:23.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.939+0000 7f609effd640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:01:23.940 INFO:teuthology.orchestra.run.smithi203.stdout:3 2024-04-04T21:01:23.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 10 librados: watch_flush enter 2024-04-04T21:01:23.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 10 librados: watch_flush exit 2024-04-04T21:01:23.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 20 client.4144.objecter shutdown clearing up homeless session... 2024-04-04T21:01:23.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 10 client.4144.objecter successfully canceled tick 2024-04-04T21:01:23.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 10 monclient: shutdown 2024-04-04T21:01:23.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:23.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 1 -- 172.21.15.203:0/1611818766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60b00a9e30 msgr2=0x7f60b0145990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:23.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 1 --2- 172.21.15.203:0/1611818766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60b00a9e30 0x7f60b0145990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f60a4033000 tx=0x7f60a40046c0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 1 -- 172.21.15.203:0/1611818766 shutdown_connections 2024-04-04T21:01:23.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 1 --2- 172.21.15.203:0/1611818766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60b0142dc0 0x7f60b0145ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 1 --2- 172.21.15.203:0/1611818766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60b00a9e30 0x7f60b0145990 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 1 --2- 172.21.15.203:0/1611818766 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60b00a9a50 0x7f60b01418a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:23.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 1 -- 172.21.15.203:0/1611818766 >> 172.21.15.203:0/1611818766 conn(0x7f60b00b8030 msgr2=0x7f60b00b8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:23.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 1 -- 172.21.15.203:0/1611818766 shutdown_connections 2024-04-04T21:01:23.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 1 -- 172.21.15.203:0/1611818766 wait complete. 2024-04-04T21:01:23.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:23.940+0000 7f60b4ac5640 1 librados: shutdown 2024-04-04T21:01:23.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new c47d034c-1db6-4332-bf24-722fb2846808 4 2024-04-04T21:01:23.976 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:23.975+0000 7f05a03b8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-04T21:01:23.993 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:23.993+0000 7f6dbac4a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-04T21:01:24.021 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:24.021+0000 7f05a03b8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-04T21:01:24.037 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:24.037+0000 7f6dbac4a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-04T21:01:24.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.077+0000 7f134049e640 10 monclient: get_monmap_and_config 2024-04-04T21:01:24.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.077+0000 7f134049e640 10 monclient: build_initial_monmap 2024-04-04T21:01:24.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.077+0000 7f134049e640 10 monclient: monmap: 2024-04-04T21:01:24.077 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:24.077 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:24.077 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:01:24.077930+0000 2024-04-04T21:01:24.077 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:01:24.077930+0000 2024-04-04T21:01:24.078 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:24.078 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:24.078 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:24.078 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:24.078 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:24.078 INFO:teuthology.orchestra.run.smithi203.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:24.078 INFO:teuthology.orchestra.run.smithi203.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:24.078 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:24.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.077+0000 7f134049e640 1 Processor -- start 2024-04-04T21:01:24.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.077+0000 7f134049e640 1 -- start start 2024-04-04T21:01:24.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.077+0000 7f134049e640 10 monclient: init 2024-04-04T21:01:24.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.078+0000 7f134049e640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:24.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.078+0000 7f134049e640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-04-04T21:01:24.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.078+0000 7f134049e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13380a9a50 addr v1:172.21.15.203:6789/0 2024-04-04T21:01:24.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.078+0000 7f134049e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f13380a9e30 addr v1:172.21.15.106:6789/0 2024-04-04T21:01:24.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.078+0000 7f134049e640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1338142f90 0x7f133814df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:24.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.078+0000 7f134049e640 10 monclient(hunting): picked mon.noname-b con 0x7f1338142f90 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:24.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.078+0000 7f134049e640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1338143420 con 0x7f13380a9e30 2024-04-04T21:01:24.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.078+0000 7f134049e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13380ab750 con 0x7f13380a9a50 2024-04-04T21:01:24.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.078+0000 7f134049e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:24.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.078+0000 7f134049e640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1338140120 con 0x7f1338142f90 2024-04-04T21:01:24.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.078+0000 7f134049e640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:24.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.078+0000 7f134049e640 10 monclient(hunting): authenticate will time out at 1324.228596s 2024-04-04T21:01:24.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.079+0000 7f133da39640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f13380a9e30 0x7f1338142880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.203:35472/0 (socket says 172.21.15.203:35472) 2024-04-04T21:01:24.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.079+0000 7f133da39640 1 -- 172.21.15.203:0/2581593790 learned_addr learned my addr 172.21.15.203:0/2581593790 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-04T21:01:24.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.079+0000 7f133ea3b640 1 --2- 172.21.15.203:0/2581593790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1338142f90 0x7f133814df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:24.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.079+0000 7f133ea3b640 10 monclient(hunting): get_auth_request con 0x7f1338142f90 auth_method 0 2024-04-04T21:01:24.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.079+0000 7f133ea3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:24.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.079+0000 7f133ea3b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:24.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.079+0000 7f133ea3b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:24.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.080+0000 7f133ea3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:24.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.080+0000 7f133ea3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:24.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.080+0000 7f133ea3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:24.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.081+0000 7f133ea3b640 10 monclient(hunting): handle_auth_done global_id 4115 payload 274 2024-04-04T21:01:24.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.081+0000 7f133ea3b640 1 -- 172.21.15.203:0/2581593790 >> v1:172.21.15.203:6789/0 conn(0x7f13380a9a50 legacy=0x7f133813f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:24.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.081+0000 7f133ea3b640 1 -- 172.21.15.203:0/2581593790 >> v1:172.21.15.106:6789/0 conn(0x7f13380a9e30 legacy=0x7f1338142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:24.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.081+0000 7f133ea3b640 10 monclient: _finish_hunting 0 2024-04-04T21:01:24.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.081+0000 7f133ea3b640 1 monclient: found mon.noname-b 2024-04-04T21:01:24.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.081+0000 7f133ea3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:24.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.081+0000 7f133ea3b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:01:24.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.081+0000 7f133ea3b640 1 -- 172.21.15.203:0/2581593790 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13380aa660 con 0x7f1338142f90 2024-04-04T21:01:24.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.083+0000 7f133ea3b640 1 --2- 172.21.15.203:0/2581593790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1338142f90 0x7f133814df00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1330009dc0 tx=0x7f133002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5b0cc397da04346 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:24.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f133d238640 1 -- 172.21.15.203:0/2581593790 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1330039070 con 0x7f1338142f90 2024-04-04T21:01:24.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f133d238640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:24.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f133d238640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:24.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f133d238640 10 monclient: dump: 2024-04-04T21:01:24.088 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:24.088 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:24.088 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.088 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.088 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:24.088 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:24.088 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:24.088 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:24.088 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:24.088 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:24.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f133d238640 10 monclient: _finish_auth 0 2024-04-04T21:01:24.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f133d238640 10 monclient: _check_auth_tickets 2024-04-04T21:01:24.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f133d238640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:24.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f133d238640 1 -- 172.21.15.203:0/2581593790 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133002cc40 con 0x7f1338142f90 2024-04-04T21:01:24.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f133d238640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:24.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f133d238640 1 -- 172.21.15.203:0/2581593790 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1330034440 con 0x7f1338142f90 2024-04-04T21:01:24.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f134049e640 5 monclient: authenticate success, global_id 4115 2024-04-04T21:01:24.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f134049e640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:24.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f134049e640 4 set_mon_vals no callback set 2024-04-04T21:01:24.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f133d238640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:24.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f133d238640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:01:24.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.087+0000 7f133d238640 10 monclient: dump: 2024-04-04T21:01:24.089 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:24.089 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:24.089 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.090 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.090 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:24.090 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:24.090 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:24.090 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:24.090 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:24.090 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:24.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.088+0000 7f134049e640 10 monclient: shutdown 2024-04-04T21:01:24.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.088+0000 7f134049e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:24.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.088+0000 7f134049e640 1 -- 172.21.15.203:0/2581593790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1338142f90 msgr2=0x7f133814df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:24.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.088+0000 7f134049e640 1 --2- 172.21.15.203:0/2581593790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1338142f90 0x7f133814df00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1330009dc0 tx=0x7f133002be90 comp rx=0 tx=0).stop 2024-04-04T21:01:24.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.088+0000 7f134049e640 1 -- 172.21.15.203:0/2581593790 shutdown_connections 2024-04-04T21:01:24.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.088+0000 7f134049e640 1 --2- 172.21.15.203:0/2581593790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1338142f90 0x7f133814df00 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:24.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.088+0000 7f134049e640 1 -- 172.21.15.203:0/2581593790 >> 172.21.15.203:0/2581593790 conn(0x7f13380b8030 msgr2=0x7f13380b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:24.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.088+0000 7f134049e640 1 -- 172.21.15.203:0/2581593790 shutdown_connections 2024-04-04T21:01:24.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.088+0000 7f134049e640 1 -- 172.21.15.203:0/2581593790 wait complete. 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.088+0000 7f134049e640 10 monclient: build_initial_monmap 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.088+0000 7f134049e640 10 monclient: monmap: 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 1 librados: starting msgr at 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 1 librados: starting objecter 2024-04-04T21:01:24.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 1 Processor -- start 2024-04-04T21:01:24.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 1 -- start start 2024-04-04T21:01:24.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 1 librados: setting wanted keys 2024-04-04T21:01:24.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 1 librados: calling monclient init 2024-04-04T21:01:24.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 10 monclient: init 2024-04-04T21:01:24.099 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:24.099 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-04T21:01:24.099 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13380a9a50 0x7f1338134600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 10 monclient(hunting): picked mon.noname-b con 0x7f13380a9a50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13380a9e30 0x7f1338143a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 10 monclient(hunting): picked mon.noname-a con 0x7f13380a9e30 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1338142f90 0x7f1338143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 10 monclient(hunting): picked mon.noname-c con 0x7f1338142f90 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133813ff30 con 0x7f13380a9e30 2024-04-04T21:01:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133813fdb0 con 0x7f13380a9a50 2024-04-04T21:01:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133814e440 con 0x7f1338142f90 2024-04-04T21:01:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.089+0000 7f134049e640 10 monclient(hunting): authenticate will time out at 1324.239766s 2024-04-04T21:01:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.090+0000 7f133ea3b640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1338142f90 0x7f1338143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:24.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.090+0000 7f133e23a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13380a9a50 0x7f1338134600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:24.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.090+0000 7f133da39640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13380a9e30 0x7f1338143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:24.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.090+0000 7f133ea3b640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1338142f90 0x7f1338143fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.203:34726/0 (socket says 172.21.15.203:34726) 2024-04-04T21:01:24.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.090+0000 7f133ea3b640 1 -- 172.21.15.203:0/2534910515 learned_addr learned my addr 172.21.15.203:0/2534910515 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-04T21:01:24.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.090+0000 7f133ea3b640 10 monclient(hunting): get_auth_request con 0x7f1338142f90 auth_method 0 2024-04-04T21:01:24.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.090+0000 7f133ea3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:24.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.090+0000 7f133ea3b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:24.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.090+0000 7f133ea3b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:24.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.090+0000 7f133e23a640 10 monclient(hunting): get_auth_request con 0x7f13380a9a50 auth_method 0 2024-04-04T21:01:24.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.090+0000 7f133e23a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:24.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.090+0000 7f133e23a640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:24.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.090+0000 7f133e23a640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:24.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.091+0000 7f133da39640 10 monclient(hunting): get_auth_request con 0x7f13380a9e30 auth_method 0 2024-04-04T21:01:24.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.091+0000 7f133da39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:24.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.091+0000 7f133da39640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:24.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.091+0000 7f133da39640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:24.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.091+0000 7f133ea3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:24.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.091+0000 7f133ea3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:24.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.091+0000 7f133ea3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:24.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.091+0000 7f133e23a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:24.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.091+0000 7f133e23a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:24.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.091+0000 7f133e23a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:24.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.091+0000 7f133da39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:24.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.091+0000 7f133da39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:24.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.091+0000 7f133da39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:24.104 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.095+0000 7f133ea3b640 10 monclient(hunting): handle_auth_done global_id 4118 payload 824 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.095+0000 7f133ea3b640 1 -- 172.21.15.203:0/2534910515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13380a9a50 msgr2=0x7f1338134600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.095+0000 7f133ea3b640 1 --2- 172.21.15.203:0/2534910515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13380a9a50 0x7f1338134600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.095+0000 7f133ea3b640 1 -- 172.21.15.203:0/2534910515 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13380a9e30 msgr2=0x7f1338143a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.095+0000 7f133ea3b640 1 --2- 172.21.15.203:0/2534910515 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13380a9e30 0x7f1338143a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.095+0000 7f133ea3b640 10 monclient: _finish_hunting 0 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.095+0000 7f133ea3b640 1 monclient: found mon.noname-c 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.095+0000 7f133ea3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.095+0000 7f133ea3b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.095+0000 7f133ea3b640 1 -- 172.21.15.203:0/2534910515 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13381459b0 con 0x7f1338142f90 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.098+0000 7f133ea3b640 1 --2- 172.21.15.203:0/2534910515 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1338142f90 0x7f1338143fa0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1330037040 tx=0x7f1330004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.098+0000 7f132effd640 10 client.?.objecter ms_handle_connect 0x7f1338142f90 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.098+0000 7f132effd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f132effd640 1 -- 172.21.15.203:0/2534910515 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1330039050 con 0x7f1338142f90 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f132effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:24.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f132effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:01:24.106 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f132effd640 10 monclient: dump: 2024-04-04T21:01:24.106 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:24.106 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:24.106 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.106 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.106 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:24.106 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:24.106 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:24.106 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:24.106 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:24.106 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:24.106 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f132effd640 10 monclient: _finish_auth 0 2024-04-04T21:01:24.106 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f132effd640 10 monclient: _check_auth_tickets 2024-04-04T21:01:24.106 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f132effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:24.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f134049e640 5 monclient: authenticate success, global_id 4118 2024-04-04T21:01:24.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f134049e640 10 monclient: _renew_subs 2024-04-04T21:01:24.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f134049e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:24.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f134049e640 1 -- 172.21.15.203:0/2534910515 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1338145dd0 con 0x7f1338142f90 2024-04-04T21:01:24.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f134049e640 10 client.4118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:24.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f134049e640 10 monclient: _renew_subs 2024-04-04T21:01:24.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f134049e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:24.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f134049e640 1 -- 172.21.15.203:0/2534910515 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13381447e0 con 0x7f1338142f90 2024-04-04T21:01:24.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.100+0000 7f134049e640 1 librados: init done 2024-04-04T21:01:24.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.101+0000 7f134049e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:24.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.101+0000 7f134049e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:24.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.101+0000 7f134049e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:24.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.101+0000 7f134049e640 1 -- 172.21.15.203:0/2534910515 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133804f050 con 0x7f1338142f90 2024-04-04T21:01:24.107 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.102+0000 7f132effd640 1 -- 172.21.15.203:0/2534910515 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1330045070 con 0x7f1338142f90 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.102+0000 7f132effd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.102+0000 7f132effd640 1 -- 172.21.15.203:0/2534910515 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133004a660 con 0x7f1338142f90 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.102+0000 7f132effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.102+0000 7f132effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.102+0000 7f132effd640 10 monclient: dump: 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:24.108 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:24.109 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.102+0000 7f133ca37640 4 set_mon_vals no callback set 2024-04-04T21:01:24.184 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:24.184+0000 7f05a03b8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-04T21:01:24.198 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:24.197+0000 7f6dbac4a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-04T21:01:24.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.293+0000 7f132effd640 1 -- 172.21.15.203:0/2534910515 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f133004a8c0 con 0x7f1338142f90 2024-04-04T21:01:24.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.293+0000 7f132effd640 1 -- 172.21.15.203:0/2534910515 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f1330054430 con 0x7f1338142f90 2024-04-04T21:01:24.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.293+0000 7f132effd640 10 client.4118.objecter ms_dispatch 0x7f1338000b90 osd_map(6..6 src has 1..6) v4 2024-04-04T21:01:24.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.293+0000 7f132effd640 3 client.4118.objecter handle_osd_map got epochs [6,6] > 0 2024-04-04T21:01:24.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.293+0000 7f132effd640 3 client.4118.objecter handle_osd_map decoding full epoch 6 2024-04-04T21:01:24.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.293+0000 7f132effd640 20 client.4118.objecter dump_active .. 0 homeless 2024-04-04T21:01:24.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.300+0000 7f132effd640 1 -- 172.21.15.203:0/2534910515 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1330040450 con 0x7f1338142f90 2024-04-04T21:01:24.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.300+0000 7f132effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:24.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.300+0000 7f132effd640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:24.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.372+0000 7f134049e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c47d034c-1db6-4332-bf24-722fb2846808", "id": 4}] 2024-04-04T21:01:24.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.372+0000 7f134049e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c47d034c-1db6-4332-bf24-722fb2846808", "id": 4}] 2024-04-04T21:01:24.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.372+0000 7f134049e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:24.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.372+0000 7f134049e640 1 -- 172.21.15.203:0/2534910515 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c47d034c-1db6-4332-bf24-722fb2846808", "id": 4} v 0) v1 -- 0x7f133813fb70 con 0x7f1338142f90 2024-04-04T21:01:24.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.506+0000 7f132effd640 1 -- 172.21.15.203:0/2534910515 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c47d034c-1db6-4332-bf24-722fb2846808", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f133004abb0 con 0x7f1338142f90 2024-04-04T21:01:24.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.506+0000 7f132effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c47d034c-1db6-4332-bf24-722fb2846808", "id": 4}] 2024-04-04T21:01:24.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.506+0000 7f132effd640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:01:24.507 INFO:teuthology.orchestra.run.smithi203.stdout:4 2024-04-04T21:01:24.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 10 librados: watch_flush enter 2024-04-04T21:01:24.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 10 librados: watch_flush exit 2024-04-04T21:01:24.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 20 client.4118.objecter shutdown clearing up homeless session... 2024-04-04T21:01:24.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 10 client.4118.objecter successfully canceled tick 2024-04-04T21:01:24.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 10 monclient: shutdown 2024-04-04T21:01:24.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:24.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 1 -- 172.21.15.203:0/2534910515 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1338142f90 msgr2=0x7f1338143fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:24.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 1 --2- 172.21.15.203:0/2534910515 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1338142f90 0x7f1338143fa0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1330037040 tx=0x7f1330004a40 comp rx=0 tx=0).stop 2024-04-04T21:01:24.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 1 -- 172.21.15.203:0/2534910515 shutdown_connections 2024-04-04T21:01:24.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 1 --2- 172.21.15.203:0/2534910515 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1338142f90 0x7f1338143fa0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:24.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 1 --2- 172.21.15.203:0/2534910515 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13380a9e30 0x7f1338143a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:24.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 1 --2- 172.21.15.203:0/2534910515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13380a9a50 0x7f1338134600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:24.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 1 -- 172.21.15.203:0/2534910515 >> 172.21.15.203:0/2534910515 conn(0x7f13380b8030 msgr2=0x7f13380b3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:24.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 1 -- 172.21.15.203:0/2534910515 shutdown_connections 2024-04-04T21:01:24.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 1 -- 172.21.15.203:0/2534910515 wait complete. 2024-04-04T21:01:24.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.507+0000 7f134049e640 1 librados: shutdown 2024-04-04T21:01:24.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 588caa62-e773-44ae-94e5-ece54fe980d2 5 2024-04-04T21:01:24.618 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.618+0000 7f679a719640 10 monclient: get_monmap_and_config 2024-04-04T21:01:24.618 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.618+0000 7f679a719640 10 monclient: build_initial_monmap 2024-04-04T21:01:24.618 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.618+0000 7f679a719640 10 monclient: monmap: 2024-04-04T21:01:24.618 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:24.618 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:24.619 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:01:24.618842+0000 2024-04-04T21:01:24.619 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:01:24.618842+0000 2024-04-04T21:01:24.619 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:24.619 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:24.619 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:24.619 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:24.619 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:24.619 INFO:teuthology.orchestra.run.smithi203.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:24.619 INFO:teuthology.orchestra.run.smithi203.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:24.619 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:24.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.618+0000 7f679a719640 1 Processor -- start 2024-04-04T21:01:24.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.618+0000 7f679a719640 1 -- start start 2024-04-04T21:01:24.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.618+0000 7f679a719640 10 monclient: init 2024-04-04T21:01:24.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.619+0000 7f679a719640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:24.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.619+0000 7f679a719640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-04-04T21:01:24.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.619+0000 7f679a719640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f67940a9a50 addr v1:172.21.15.106:6789/0 2024-04-04T21:01:24.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.619+0000 7f679a719640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67940a9e30 addr v1:172.21.15.203:6789/0 2024-04-04T21:01:24.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.619+0000 7f679a719640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6794142f90 0x7f679414df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.619+0000 7f679a719640 10 monclient(hunting): picked mon.noname-b con 0x7f6794142f90 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.619+0000 7f679a719640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6794143420 con 0x7f67940a9a50 2024-04-04T21:01:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.619+0000 7f679a719640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67940ab750 con 0x7f67940a9e30 2024-04-04T21:01:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.619+0000 7f679a719640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.619+0000 7f679a719640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6794140120 con 0x7f6794142f90 2024-04-04T21:01:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.619+0000 7f679a719640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.619+0000 7f679a719640 10 monclient(hunting): authenticate will time out at 1324.769485s 2024-04-04T21:01:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.620+0000 7f67937fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f67940a9e30 0x7f6794142880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45530/0 (socket says 172.21.15.203:45530) 2024-04-04T21:01:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.620+0000 7f67937fe640 1 -- 172.21.15.203:0/1606399034 learned_addr learned my addr 172.21.15.203:0/1606399034 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-04T21:01:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.620+0000 7f6798cb6640 1 --2- 172.21.15.203:0/1606399034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6794142f90 0x7f679414df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.620+0000 7f6798cb6640 10 monclient(hunting): get_auth_request con 0x7f6794142f90 auth_method 0 2024-04-04T21:01:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.620+0000 7f6798cb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.620+0000 7f6798cb6640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.620+0000 7f6798cb6640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:24.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.621+0000 7f6798cb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:24.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.621+0000 7f6798cb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:24.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.621+0000 7f6798cb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:24.624 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.622+0000 7f6792ffd640 1 -- 172.21.15.203:0/1606399034 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263948127 0 0) 0x7f67940ab750 con 0x7f67940a9e30 2024-04-04T21:01:24.624 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.622+0000 7f6792ffd640 10 monclient(hunting): _negotiate 2024-04-04T21:01:24.624 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.622+0000 7f6792ffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:24.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.622+0000 7f6792ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:24.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.622+0000 7f6792ffd640 10 monclient(hunting): my global_id is 4153 2024-04-04T21:01:24.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.622+0000 7f6792ffd640 1 -- 172.21.15.203:0/1606399034 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f677c003520 con 0x7f67940a9e30 2024-04-04T21:01:24.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.622+0000 7f6798cb6640 10 monclient(hunting): handle_auth_done global_id 4121 payload 274 2024-04-04T21:01:24.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.622+0000 7f6798cb6640 1 -- 172.21.15.203:0/1606399034 >> v1:172.21.15.203:6789/0 conn(0x7f67940a9e30 legacy=0x7f6794142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:24.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.622+0000 7f6798cb6640 1 -- 172.21.15.203:0/1606399034 >> v1:172.21.15.106:6789/0 conn(0x7f67940a9a50 legacy=0x7f679413f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:24.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.622+0000 7f6798cb6640 10 monclient: _finish_hunting 0 2024-04-04T21:01:24.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.622+0000 7f6798cb6640 1 monclient: found mon.noname-b 2024-04-04T21:01:24.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.622+0000 7f6798cb6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:24.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.622+0000 7f6798cb6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:01:24.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.622+0000 7f6798cb6640 1 -- 172.21.15.203:0/1606399034 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67940aa660 con 0x7f6794142f90 2024-04-04T21:01:24.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.624+0000 7f6798cb6640 1 --2- 172.21.15.203:0/1606399034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6794142f90 0x7f679414df00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f678c009dc0 tx=0x7f678c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a990ae728d19bfb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:24.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f6792ffd640 1 -- 172.21.15.203:0/1606399034 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c039070 con 0x7f6794142f90 2024-04-04T21:01:24.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f6792ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:24.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f6792ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:24.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f6792ffd640 10 monclient: dump: 2024-04-04T21:01:24.628 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:24.628 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:24.628 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.628 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.628 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f6792ffd640 10 monclient: _finish_auth 0 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f6792ffd640 10 monclient: _check_auth_tickets 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f6792ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f6792ffd640 1 -- 172.21.15.203:0/1606399034 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678c02cc40 con 0x7f6794142f90 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f679a719640 5 monclient: authenticate success, global_id 4121 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f679a719640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f6792ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f6792ffd640 1 -- 172.21.15.203:0/1606399034 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c034440 con 0x7f6794142f90 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f679a719640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:24.629 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f679a719640 4 set_mon_vals no callback set 2024-04-04T21:01:24.630 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f6792ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:24.630 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f6792ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:01:24.630 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.628+0000 7f6792ffd640 10 monclient: dump: 2024-04-04T21:01:24.630 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:24.630 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:24.630 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.630 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.630 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:24.630 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:24.630 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:24.630 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:24.631 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:24.631 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:24.635 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.629+0000 7f679a719640 10 monclient: shutdown 2024-04-04T21:01:24.635 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.629+0000 7f679a719640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:24.635 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.629+0000 7f679a719640 1 -- 172.21.15.203:0/1606399034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6794142f90 msgr2=0x7f679414df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:24.635 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.629+0000 7f679a719640 1 --2- 172.21.15.203:0/1606399034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6794142f90 0x7f679414df00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f678c009dc0 tx=0x7f678c02be90 comp rx=0 tx=0).stop 2024-04-04T21:01:24.635 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.629+0000 7f679a719640 1 -- 172.21.15.203:0/1606399034 shutdown_connections 2024-04-04T21:01:24.635 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.629+0000 7f679a719640 1 --2- 172.21.15.203:0/1606399034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6794142f90 0x7f679414df00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:24.635 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.629+0000 7f679a719640 1 -- 172.21.15.203:0/1606399034 >> 172.21.15.203:0/1606399034 conn(0x7f67940b8030 msgr2=0x7f67940b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:24.635 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.629+0000 7f679a719640 1 -- 172.21.15.203:0/1606399034 shutdown_connections 2024-04-04T21:01:24.635 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.629+0000 7f679a719640 1 -- 172.21.15.203:0/1606399034 wait complete. 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.629+0000 7f679a719640 10 monclient: build_initial_monmap 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.629+0000 7f679a719640 10 monclient: monmap: 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 1 librados: starting msgr at 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 1 librados: starting objecter 2024-04-04T21:01:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 1 Processor -- start 2024-04-04T21:01:24.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 1 -- start start 2024-04-04T21:01:24.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 1 librados: setting wanted keys 2024-04-04T21:01:24.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 1 librados: calling monclient init 2024-04-04T21:01:24.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 10 monclient: init 2024-04-04T21:01:24.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:24.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:01:24.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f67940a9a50 0x7f6794145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:24.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 10 monclient(hunting): picked mon.noname-c con 0x7f67940a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:24.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67940a9e30 0x7f67941385e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:24.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 10 monclient(hunting): picked mon.noname-b con 0x7f67940a9e30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:24.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6794142f90 0x7f6794138b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:24.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 10 monclient(hunting): picked mon.noname-a con 0x7f6794142f90 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:24.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:24.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679413ff30 con 0x7f6794142f90 2024-04-04T21:01:24.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:24.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679413fdb0 con 0x7f67940a9e30 2024-04-04T21:01:24.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:24.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679414e440 con 0x7f67940a9a50 2024-04-04T21:01:24.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:24.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.630+0000 7f679a719640 10 monclient(hunting): authenticate will time out at 1324.780892s 2024-04-04T21:01:24.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f67937fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67940a9e30 0x7f67941385e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:24.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f6798cb6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6794142f90 0x7f6794138b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:24.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f6793fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f67940a9a50 0x7f6794145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:24.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f6793fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f67940a9a50 0x7f6794145830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.203:34736/0 (socket says 172.21.15.203:34736) 2024-04-04T21:01:24.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f6793fff640 1 -- 172.21.15.203:0/794832839 learned_addr learned my addr 172.21.15.203:0/794832839 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-04T21:01:24.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f6793fff640 10 monclient(hunting): get_auth_request con 0x7f67940a9a50 auth_method 0 2024-04-04T21:01:24.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f6793fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:24.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f6793fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:24.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f6793fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:24.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f67937fe640 10 monclient(hunting): get_auth_request con 0x7f67940a9e30 auth_method 0 2024-04-04T21:01:24.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f67937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:24.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f67937fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:24.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f67937fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:24.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f6798cb6640 10 monclient(hunting): get_auth_request con 0x7f6794142f90 auth_method 0 2024-04-04T21:01:24.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f6798cb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:24.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f6798cb6640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:24.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.631+0000 7f6798cb6640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:24.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.632+0000 7f67937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:24.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.632+0000 7f67937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:24.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.632+0000 7f67937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:24.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.632+0000 7f6793fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.632+0000 7f6793fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.632+0000 7f6793fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.632+0000 7f6798cb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.632+0000 7f6798cb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.632+0000 7f6798cb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.634+0000 7f67937fe640 10 monclient(hunting): handle_auth_done global_id 4156 payload 824 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.634+0000 7f67937fe640 1 -- 172.21.15.203:0/794832839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f67940a9a50 msgr2=0x7f6794145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.634+0000 7f67937fe640 1 --2- 172.21.15.203:0/794832839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f67940a9a50 0x7f6794145830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.635+0000 7f67937fe640 1 -- 172.21.15.203:0/794832839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6794142f90 msgr2=0x7f6794138b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.635+0000 7f67937fe640 1 --2- 172.21.15.203:0/794832839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6794142f90 0x7f6794138b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.635+0000 7f67937fe640 10 monclient: _finish_hunting 0 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.635+0000 7f67937fe640 1 monclient: found mon.noname-b 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.635+0000 7f67937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.635+0000 7f67937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:01:24.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.635+0000 7f67937fe640 1 -- 172.21.15.203:0/794832839 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679413b8e0 con 0x7f67940a9e30 2024-04-04T21:01:24.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.637+0000 7f67937fe640 1 --2- 172.21.15.203:0/794832839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67940a9e30 0x7f67941385e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f678800c460 tx=0x7f678800c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:24.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.637+0000 7f6790ff9640 10 client.?.objecter ms_handle_connect 0x7f67940a9e30 2024-04-04T21:01:24.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.637+0000 7f6790ff9640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:24.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f6790ff9640 1 -- 172.21.15.203:0/794832839 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6788017070 con 0x7f67940a9e30 2024-04-04T21:01:24.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f6790ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:24.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f6790ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:24.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f6790ff9640 10 monclient: dump: 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f6790ff9640 10 monclient: _finish_auth 0 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f6790ff9640 10 monclient: _check_auth_tickets 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f6790ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f6790ff9640 1 -- 172.21.15.203:0/794832839 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6788005ce0 con 0x7f67940a9e30 2024-04-04T21:01:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f679a719640 5 monclient: authenticate success, global_id 4156 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f6790ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f6790ff9640 1 -- 172.21.15.203:0/794832839 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67880072b0 con 0x7f67940a9e30 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f6790ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f6790ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f6790ff9640 10 monclient: dump: 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:24.645 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:24.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f67927fc640 4 set_mon_vals no callback set 2024-04-04T21:01:24.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f679a719640 10 monclient: _renew_subs 2024-04-04T21:01:24.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f679a719640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:24.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f679a719640 1 -- 172.21.15.203:0/794832839 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679413bbc0 con 0x7f67940a9e30 2024-04-04T21:01:24.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f679a719640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:24.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f679a719640 10 monclient: _renew_subs 2024-04-04T21:01:24.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f679a719640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:24.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f679a719640 1 -- 172.21.15.203:0/794832839 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6794139260 con 0x7f67940a9e30 2024-04-04T21:01:24.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.640+0000 7f679a719640 1 librados: init done 2024-04-04T21:01:24.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.641+0000 7f679a719640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:24.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.641+0000 7f679a719640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:24.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.641+0000 7f679a719640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:24.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.641+0000 7f679a719640 1 -- 172.21.15.203:0/794832839 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679404f050 con 0x7f67940a9e30 2024-04-04T21:01:24.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.644+0000 7f6790ff9640 1 -- 172.21.15.203:0/794832839 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f678801a070 con 0x7f67940a9e30 2024-04-04T21:01:24.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.644+0000 7f6790ff9640 1 -- 172.21.15.203:0/794832839 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f678801b930 con 0x7f67940a9e30 2024-04-04T21:01:24.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.644+0000 7f6790ff9640 10 client.4156.objecter ms_dispatch 0x7f6794000b90 osd_map(7..7 src has 1..7) v4 2024-04-04T21:01:24.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.644+0000 7f6790ff9640 3 client.4156.objecter handle_osd_map got epochs [7,7] > 0 2024-04-04T21:01:24.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.644+0000 7f6790ff9640 3 client.4156.objecter handle_osd_map decoding full epoch 7 2024-04-04T21:01:24.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.644+0000 7f6790ff9640 20 client.4156.objecter dump_active .. 0 homeless 2024-04-04T21:01:24.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.739+0000 7f6790ff9640 1 -- 172.21.15.203:0/794832839 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6788021020 con 0x7f67940a9e30 2024-04-04T21:01:24.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.739+0000 7f6790ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:24.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.739+0000 7f6790ff9640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:24.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.808+0000 7f679a719640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "588caa62-e773-44ae-94e5-ece54fe980d2", "id": 5}] 2024-04-04T21:01:24.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.808+0000 7f679a719640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "588caa62-e773-44ae-94e5-ece54fe980d2", "id": 5}] 2024-04-04T21:01:24.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.808+0000 7f679a719640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:24.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.808+0000 7f679a719640 1 -- 172.21.15.203:0/794832839 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "588caa62-e773-44ae-94e5-ece54fe980d2", "id": 5} v 0) v1 -- 0x7f67940b79e0 con 0x7f67940a9e30 2024-04-04T21:01:24.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.920+0000 7f6790ff9640 1 -- 172.21.15.203:0/794832839 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "588caa62-e773-44ae-94e5-ece54fe980d2", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f679413b8e0 con 0x7f67940a9e30 2024-04-04T21:01:24.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.920+0000 7f6790ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "588caa62-e773-44ae-94e5-ece54fe980d2", "id": 5}] 2024-04-04T21:01:24.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.920+0000 7f6790ff9640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:01:24.921 INFO:teuthology.orchestra.run.smithi203.stdout:5 2024-04-04T21:01:24.922 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.921+0000 7f679a719640 10 librados: watch_flush enter 2024-04-04T21:01:24.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.921+0000 7f679a719640 10 librados: watch_flush exit 2024-04-04T21:01:24.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.921+0000 7f679a719640 20 client.4156.objecter shutdown clearing up homeless session... 2024-04-04T21:01:24.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.921+0000 7f679a719640 10 client.4156.objecter successfully canceled tick 2024-04-04T21:01:24.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.921+0000 7f679a719640 10 monclient: shutdown 2024-04-04T21:01:24.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.921+0000 7f679a719640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:24.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.921+0000 7f679a719640 1 -- 172.21.15.203:0/794832839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67940a9e30 msgr2=0x7f67941385e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:24.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.921+0000 7f679a719640 1 --2- 172.21.15.203:0/794832839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67940a9e30 0x7f67941385e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f678800c460 tx=0x7f678800c920 comp rx=0 tx=0).stop 2024-04-04T21:01:24.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.921+0000 7f679a719640 1 -- 172.21.15.203:0/794832839 shutdown_connections 2024-04-04T21:01:24.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.921+0000 7f679a719640 1 --2- 172.21.15.203:0/794832839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6794142f90 0x7f6794138b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:24.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.921+0000 7f679a719640 1 --2- 172.21.15.203:0/794832839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67940a9e30 0x7f67941385e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:24.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.921+0000 7f679a719640 1 --2- 172.21.15.203:0/794832839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f67940a9a50 0x7f6794145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:24.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.921+0000 7f679a719640 1 -- 172.21.15.203:0/794832839 >> 172.21.15.203:0/794832839 conn(0x7f67940b8030 msgr2=0x7f67940b5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:24.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.922+0000 7f679a719640 1 -- 172.21.15.203:0/794832839 shutdown_connections 2024-04-04T21:01:24.924 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.922+0000 7f679a719640 1 -- 172.21.15.203:0/794832839 wait complete. 2024-04-04T21:01:24.924 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:24.922+0000 7f679a719640 1 librados: shutdown 2024-04-04T21:01:24.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new ef776e12-7c80-4095-b173-5b2d24996f95 6 2024-04-04T21:01:25.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.035+0000 7f8489e82640 10 monclient: get_monmap_and_config 2024-04-04T21:01:25.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.035+0000 7f8489e82640 10 monclient: build_initial_monmap 2024-04-04T21:01:25.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.035+0000 7f8489e82640 10 monclient: monmap: 2024-04-04T21:01:25.036 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:25.036 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.036 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:01:25.036460+0000 2024-04-04T21:01:25.036 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:01:25.036460+0000 2024-04-04T21:01:25.036 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:25.036 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:25.036 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:25.036 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:25.036 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:25.036 INFO:teuthology.orchestra.run.smithi203.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:25.036 INFO:teuthology.orchestra.run.smithi203.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:25.037 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:25.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.036+0000 7f8489e82640 1 Processor -- start 2024-04-04T21:01:25.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.036+0000 7f8489e82640 1 -- start start 2024-04-04T21:01:25.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.036+0000 7f8489e82640 10 monclient: init 2024-04-04T21:01:25.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.037+0000 7f8489e82640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:25.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.037+0000 7f8489e82640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-04-04T21:01:25.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.037+0000 7f8489e82640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84840a9a50 0x7f84840a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.037+0000 7f8489e82640 10 monclient(hunting): picked mon.noname-b con 0x7f84840a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:25.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.037+0000 7f8489e82640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f848413f6a0 addr v1:172.21.15.106:6789/0 2024-04-04T21:01:25.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.037+0000 7f8489e82640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f848413fac0 addr v1:172.21.15.203:6789/0 2024-04-04T21:01:25.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.037+0000 7f8489e82640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84840ab750 con 0x7f848413f6a0 2024-04-04T21:01:25.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.037+0000 7f8489e82640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8484140120 con 0x7f848413fac0 2024-04-04T21:01:25.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.037+0000 7f8489e82640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.037+0000 7f8489e82640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f848413ff50 con 0x7f84840a9a50 2024-04-04T21:01:25.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.037+0000 7f8489e82640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:25.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.037+0000 7f8489e82640 10 monclient(hunting): authenticate will time out at 1325.187699s 2024-04-04T21:01:25.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.038+0000 7f8482ffd640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f848413f6a0 0x7f8484142880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.203:35496/0 (socket says 172.21.15.203:35496) 2024-04-04T21:01:25.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.038+0000 7f8482ffd640 1 -- 172.21.15.203:0/412770276 learned_addr learned my addr 172.21.15.203:0/412770276 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-04T21:01:25.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.038+0000 7f84837fe640 1 --2- 172.21.15.203:0/412770276 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84840a9a50 0x7f84840a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:25.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.039+0000 7f84837fe640 10 monclient(hunting): get_auth_request con 0x7f84840a9a50 auth_method 0 2024-04-04T21:01:25.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.039+0000 7f84837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.039+0000 7f84837fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.039+0000 7f84837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.039+0000 7f84837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.039+0000 7f84837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.039+0000 7f84837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.041+0000 7f84827fc640 1 -- 172.21.15.203:0/412770276 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496958806 0 0) 0x7f84840ab750 con 0x7f848413f6a0 2024-04-04T21:01:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.041+0000 7f84827fc640 10 monclient(hunting): _negotiate 2024-04-04T21:01:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.041+0000 7f84827fc640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.041+0000 7f84827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.041+0000 7f84827fc640 10 monclient(hunting): my global_id is 4149 2024-04-04T21:01:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.041+0000 7f84827fc640 1 -- 172.21.15.203:0/412770276 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f846c003520 con 0x7f848413f6a0 2024-04-04T21:01:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.041+0000 7f84827fc640 1 -- 172.21.15.203:0/412770276 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184159509 0 0) 0x7f8484140120 con 0x7f848413fac0 2024-04-04T21:01:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.041+0000 7f84827fc640 10 monclient(hunting): _negotiate 2024-04-04T21:01:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.041+0000 7f84827fc640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.041+0000 7f84827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.041+0000 7f84827fc640 10 monclient(hunting): my global_id is 4159 2024-04-04T21:01:25.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.041+0000 7f84827fc640 1 -- 172.21.15.203:0/412770276 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84840ab750 con 0x7f848413fac0 2024-04-04T21:01:25.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.042+0000 7f84837fe640 10 monclient(hunting): handle_auth_done global_id 4127 payload 274 2024-04-04T21:01:25.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.042+0000 7f84837fe640 1 -- 172.21.15.203:0/412770276 >> v1:172.21.15.203:6789/0 conn(0x7f848413fac0 legacy=0x7f848414cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:25.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.042+0000 7f84837fe640 1 -- 172.21.15.203:0/412770276 >> v1:172.21.15.106:6789/0 conn(0x7f848413f6a0 legacy=0x7f8484142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:25.044 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.042+0000 7f84837fe640 10 monclient: _finish_hunting 0 2024-04-04T21:01:25.044 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.042+0000 7f84837fe640 1 monclient: found mon.noname-b 2024-04-04T21:01:25.044 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.042+0000 7f84837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:25.044 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.042+0000 7f84837fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:01:25.044 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.042+0000 7f84837fe640 1 -- 172.21.15.203:0/412770276 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84840aa660 con 0x7f84840a9a50 2024-04-04T21:01:25.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.045+0000 7f84837fe640 1 --2- 172.21.15.203:0/412770276 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84840a9a50 0x7f84840a9e30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f84740097d0 tx=0x7f847402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8c0acca80205cbc server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:25.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f84827fc640 1 -- 172.21.15.203:0/412770276 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474035020 con 0x7f84840a9a50 2024-04-04T21:01:25.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f84827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:25.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f84827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:25.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f84827fc640 10 monclient: dump: 2024-04-04T21:01:25.051 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:25.051 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.051 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.052 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.052 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:25.052 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:25.052 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:25.052 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:25.052 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:25.052 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:25.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f84827fc640 10 monclient: _finish_auth 0 2024-04-04T21:01:25.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f84827fc640 10 monclient: _check_auth_tickets 2024-04-04T21:01:25.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f84827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:25.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f84827fc640 1 -- 172.21.15.203:0/412770276 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8474031800 con 0x7f84840a9a50 2024-04-04T21:01:25.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f84827fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:25.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f84827fc640 1 -- 172.21.15.203:0/412770276 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474031bd0 con 0x7f84840a9a50 2024-04-04T21:01:25.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f84827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f84827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f84827fc640 10 monclient: dump: 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f8489e82640 5 monclient: authenticate success, global_id 4127 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f8489e82640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:25.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.049+0000 7f8489e82640 4 set_mon_vals no callback set 2024-04-04T21:01:25.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.050+0000 7f8489e82640 10 monclient: shutdown 2024-04-04T21:01:25.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.050+0000 7f8489e82640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:25.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.050+0000 7f8489e82640 1 -- 172.21.15.203:0/412770276 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84840a9a50 msgr2=0x7f84840a9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:25.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.050+0000 7f8489e82640 1 --2- 172.21.15.203:0/412770276 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84840a9a50 0x7f84840a9e30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f84740097d0 tx=0x7f847402b8a0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.050+0000 7f8489e82640 1 -- 172.21.15.203:0/412770276 shutdown_connections 2024-04-04T21:01:25.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.050+0000 7f8489e82640 1 --2- 172.21.15.203:0/412770276 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84840a9a50 0x7f84840a9e30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.050+0000 7f8489e82640 1 -- 172.21.15.203:0/412770276 >> 172.21.15.203:0/412770276 conn(0x7f84840b8030 msgr2=0x7f84840b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:25.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.050+0000 7f8489e82640 1 -- 172.21.15.203:0/412770276 shutdown_connections 2024-04-04T21:01:25.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.050+0000 7f8489e82640 1 -- 172.21.15.203:0/412770276 wait complete. 2024-04-04T21:01:25.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.050+0000 7f8489e82640 10 monclient: build_initial_monmap 2024-04-04T21:01:25.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.050+0000 7f8489e82640 10 monclient: monmap: 2024-04-04T21:01:25.054 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:25.054 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:25.054 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-04-04T21:01:25.055 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-04-04T21:01:25.055 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:25.055 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:25.055 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:25.055 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:25.055 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:25.055 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:25.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.051+0000 7f8489e82640 1 librados: starting msgr at 2024-04-04T21:01:25.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.051+0000 7f8489e82640 1 librados: starting objecter 2024-04-04T21:01:25.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.051+0000 7f8489e82640 1 Processor -- start 2024-04-04T21:01:25.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.051+0000 7f8489e82640 1 -- start start 2024-04-04T21:01:25.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.051+0000 7f8489e82640 1 librados: setting wanted keys 2024-04-04T21:01:25.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.051+0000 7f8489e82640 1 librados: calling monclient init 2024-04-04T21:01:25.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.051+0000 7f8489e82640 10 monclient: init 2024-04-04T21:01:25.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.051+0000 7f8489e82640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:25.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.051+0000 7f8489e82640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-04T21:01:25.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8489e82640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84840a9a50 0x7f8484138fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8489e82640 10 monclient(hunting): picked mon.noname-b con 0x7f84840a9a50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8489e82640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f848413f6a0 0x7f84841cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8489e82640 10 monclient(hunting): picked mon.noname-c con 0x7f848413f6a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8489e82640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f848413fac0 0x7f8484138270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8489e82640 10 monclient(hunting): picked mon.noname-a con 0x7f848413fac0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8489e82640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8489e82640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84841433e0 con 0x7f848413fac0 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8489e82640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8489e82640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8484143260 con 0x7f84840a9a50 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8489e82640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8489e82640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84840aa370 con 0x7f848413f6a0 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8489e82640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8489e82640 10 monclient(hunting): authenticate will time out at 1325.201983s 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f84837fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84840a9a50 0x7f8484138fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8482ffd640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f848413f6a0 0x7f84841cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:25.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f8483fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f848413fac0 0x7f8484138270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:25.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f84837fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84840a9a50 0x7f8484138fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:44708/0 (socket says 172.21.15.203:44708) 2024-04-04T21:01:25.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f84837fe640 1 -- 172.21.15.203:0/368588277 learned_addr learned my addr 172.21.15.203:0/368588277 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-04T21:01:25.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f84837fe640 10 monclient(hunting): get_auth_request con 0x7f84840a9a50 auth_method 0 2024-04-04T21:01:25.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f84837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:25.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f84837fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.052+0000 7f84837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f8483fff640 10 monclient(hunting): get_auth_request con 0x7f848413fac0 auth_method 0 2024-04-04T21:01:25.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f8483fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:25.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f8483fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f8483fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f8482ffd640 10 monclient(hunting): get_auth_request con 0x7f848413f6a0 auth_method 0 2024-04-04T21:01:25.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f8482ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:25.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f8482ffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f8482ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f84837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:25.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f84837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:25.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f84837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:25.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f8483fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:25.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f8483fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:25.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f8483fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:25.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f8482ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:25.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f8482ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:25.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.053+0000 7f8482ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:25.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.056+0000 7f84837fe640 10 monclient(hunting): handle_auth_done global_id 4162 payload 824 2024-04-04T21:01:25.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.056+0000 7f84837fe640 1 -- 172.21.15.203:0/368588277 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f848413f6a0 msgr2=0x7f84841cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:25.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.056+0000 7f84837fe640 1 --2- 172.21.15.203:0/368588277 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f848413f6a0 0x7f84841cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.056+0000 7f84837fe640 1 -- 172.21.15.203:0/368588277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f848413fac0 msgr2=0x7f8484138270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:25.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.056+0000 7f84837fe640 1 --2- 172.21.15.203:0/368588277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f848413fac0 0x7f8484138270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.056+0000 7f84837fe640 10 monclient: _finish_hunting 0 2024-04-04T21:01:25.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.056+0000 7f84837fe640 1 monclient: found mon.noname-b 2024-04-04T21:01:25.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.056+0000 7f84837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:25.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.056+0000 7f84837fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:01:25.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.056+0000 7f84837fe640 1 -- 172.21.15.203:0/368588277 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84841cf4b0 con 0x7f84840a9a50 2024-04-04T21:01:25.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.058+0000 7f84837fe640 1 --2- 172.21.15.203:0/368588277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84840a9a50 0x7f8484138fc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f847402beb0 tx=0x7f84740046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:25.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.058+0000 7f846bfff640 10 client.?.objecter ms_handle_connect 0x7f84840a9a50 2024-04-04T21:01:25.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.058+0000 7f846bfff640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:25.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f846bfff640 1 -- 172.21.15.203:0/368588277 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474035040 con 0x7f84840a9a50 2024-04-04T21:01:25.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f846bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:25.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f846bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:25.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f846bfff640 10 monclient: dump: 2024-04-04T21:01:25.066 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:25.066 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.066 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.066 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.066 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:25.066 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:25.066 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:25.067 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:25.067 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:25.067 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:25.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f846bfff640 10 monclient: _finish_auth 0 2024-04-04T21:01:25.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f846bfff640 10 monclient: _check_auth_tickets 2024-04-04T21:01:25.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f846bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:25.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f846bfff640 1 -- 172.21.15.203:0/368588277 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847403ad00 con 0x7f84840a9a50 2024-04-04T21:01:25.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f846bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:25.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f846bfff640 1 -- 172.21.15.203:0/368588277 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84740466c0 con 0x7f84840a9a50 2024-04-04T21:01:25.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f8489e82640 5 monclient: authenticate success, global_id 4162 2024-04-04T21:01:25.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f8489e82640 10 monclient: _renew_subs 2024-04-04T21:01:25.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f8489e82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:25.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f8489e82640 1 -- 172.21.15.203:0/368588277 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84841ce340 con 0x7f84840a9a50 2024-04-04T21:01:25.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f846bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f846bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f846bfff640 10 monclient: dump: 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f8481ffb640 4 set_mon_vals no callback set 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f8489e82640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f8489e82640 10 monclient: _renew_subs 2024-04-04T21:01:25.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f8489e82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:25.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f8489e82640 1 -- 172.21.15.203:0/368588277 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84841387b0 con 0x7f84840a9a50 2024-04-04T21:01:25.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.061+0000 7f8489e82640 1 librados: init done 2024-04-04T21:01:25.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.063+0000 7f8489e82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:25.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.063+0000 7f8489e82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:25.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.063+0000 7f8489e82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:25.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.063+0000 7f8489e82640 1 -- 172.21.15.203:0/368588277 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848404f050 con 0x7f84840a9a50 2024-04-04T21:01:25.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.065+0000 7f846bfff640 1 -- 172.21.15.203:0/368588277 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8474038020 con 0x7f84840a9a50 2024-04-04T21:01:25.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.065+0000 7f846bfff640 1 -- 172.21.15.203:0/368588277 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f847403bd80 con 0x7f84840a9a50 2024-04-04T21:01:25.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.065+0000 7f846bfff640 10 client.4162.objecter ms_dispatch 0x7f8484000b90 osd_map(8..8 src has 1..8) v4 2024-04-04T21:01:25.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.065+0000 7f846bfff640 3 client.4162.objecter handle_osd_map got epochs [8,8] > 0 2024-04-04T21:01:25.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.065+0000 7f846bfff640 3 client.4162.objecter handle_osd_map decoding full epoch 8 2024-04-04T21:01:25.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.065+0000 7f846bfff640 20 client.4162.objecter dump_active .. 0 homeless 2024-04-04T21:01:25.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.167+0000 7f846bfff640 1 -- 172.21.15.203:0/368588277 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f847403b750 con 0x7f84840a9a50 2024-04-04T21:01:25.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.167+0000 7f846bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:25.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.167+0000 7f846bfff640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:25.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.257+0000 7f8489e82640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ef776e12-7c80-4095-b173-5b2d24996f95", "id": 6}] 2024-04-04T21:01:25.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.257+0000 7f8489e82640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ef776e12-7c80-4095-b173-5b2d24996f95", "id": 6}] 2024-04-04T21:01:25.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.257+0000 7f8489e82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:25.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.257+0000 7f8489e82640 1 -- 172.21.15.203:0/368588277 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ef776e12-7c80-4095-b173-5b2d24996f95", "id": 6} v 0) v1 -- 0x7f8484067ed0 con 0x7f84840a9a50 2024-04-04T21:01:25.264 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:25.262+0000 7f6dbac4a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-04T21:01:25.306 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:25.306+0000 7f05a03b8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-04T21:01:25.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.319+0000 7f846bfff640 1 -- 172.21.15.203:0/368588277 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ef776e12-7c80-4095-b173-5b2d24996f95", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f847403ba20 con 0x7f84840a9a50 2024-04-04T21:01:25.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.319+0000 7f846bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ef776e12-7c80-4095-b173-5b2d24996f95", "id": 6}] 2024-04-04T21:01:25.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.319+0000 7f846bfff640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:01:25.319 INFO:teuthology.orchestra.run.smithi203.stdout:6 2024-04-04T21:01:25.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.319+0000 7f8489e82640 10 librados: watch_flush enter 2024-04-04T21:01:25.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.319+0000 7f8489e82640 10 librados: watch_flush exit 2024-04-04T21:01:25.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.319+0000 7f8489e82640 20 client.4162.objecter shutdown clearing up homeless session... 2024-04-04T21:01:25.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.319+0000 7f8489e82640 10 client.4162.objecter successfully canceled tick 2024-04-04T21:01:25.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.320+0000 7f8489e82640 10 monclient: shutdown 2024-04-04T21:01:25.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.320+0000 7f8489e82640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:25.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.320+0000 7f8489e82640 1 -- 172.21.15.203:0/368588277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84840a9a50 msgr2=0x7f8484138fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:25.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.320+0000 7f8489e82640 1 --2- 172.21.15.203:0/368588277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84840a9a50 0x7f8484138fc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f847402beb0 tx=0x7f84740046c0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.320+0000 7f8489e82640 1 -- 172.21.15.203:0/368588277 shutdown_connections 2024-04-04T21:01:25.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.320+0000 7f8489e82640 1 --2- 172.21.15.203:0/368588277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f848413fac0 0x7f8484138270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.320+0000 7f8489e82640 1 --2- 172.21.15.203:0/368588277 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f848413f6a0 0x7f84841cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.322 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.320+0000 7f8489e82640 1 --2- 172.21.15.203:0/368588277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84840a9a50 0x7f8484138fc0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.322 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.320+0000 7f8489e82640 1 -- 172.21.15.203:0/368588277 >> 172.21.15.203:0/368588277 conn(0x7f84840b8030 msgr2=0x7f84840b8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:25.322 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.320+0000 7f8489e82640 1 -- 172.21.15.203:0/368588277 shutdown_connections 2024-04-04T21:01:25.322 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.320+0000 7f8489e82640 1 -- 172.21.15.203:0/368588277 wait complete. 2024-04-04T21:01:25.322 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.320+0000 7f8489e82640 1 librados: shutdown 2024-04-04T21:01:25.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 53aafc58-6463-4aeb-8395-e14fcfcee5de 7 2024-04-04T21:01:25.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.430+0000 7fd36790b640 10 monclient: get_monmap_and_config 2024-04-04T21:01:25.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.430+0000 7fd36790b640 10 monclient: build_initial_monmap 2024-04-04T21:01:25.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.430+0000 7fd36790b640 10 monclient: monmap: 2024-04-04T21:01:25.431 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:25.431 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.431 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:01:25.431490+0000 2024-04-04T21:01:25.431 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:01:25.431490+0000 2024-04-04T21:01:25.431 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:25.431 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:25.431 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:25.432 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:25.432 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:25.432 INFO:teuthology.orchestra.run.smithi203.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:25.432 INFO:teuthology.orchestra.run.smithi203.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:25.432 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:25.432 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.431+0000 7fd36790b640 1 Processor -- start 2024-04-04T21:01:25.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.431+0000 7fd36790b640 1 -- start start 2024-04-04T21:01:25.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.431+0000 7fd36790b640 10 monclient: init 2024-04-04T21:01:25.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd36790b640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:25.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd36790b640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-04T21:01:25.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd36790b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3600a9a50 addr v1:172.21.15.203:6789/0 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd36790b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd3600a9e30 0x7fd360142880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd36790b640 10 monclient(hunting): picked mon.noname-c con 0x7fd3600a9e30 addr v2:172.21.15.203:3300/0 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd36790b640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd360142dc0 0x7fd36014cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd36790b640 10 monclient(hunting): picked mon.noname-a con 0x7fd360142dc0 addr v2:172.21.15.106:3300/0 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd36790b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3600ab750 con 0x7fd3600a9a50 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd36790b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd36790b640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd360140120 con 0x7fd360142dc0 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd36790b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd36790b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3600aa660 con 0x7fd3600a9e30 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd36790b640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd36790b640 10 monclient(hunting): authenticate will time out at 1325.582140s 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd365ea8640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd360142dc0 0x7fd36014cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.432+0000 7fd364ea6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd3600a9e30 0x7fd360142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.433+0000 7fd365ea8640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd360142dc0 0x7fd36014cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.203:56542/0 (socket says 172.21.15.203:56542) 2024-04-04T21:01:25.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.433+0000 7fd365ea8640 1 -- 172.21.15.203:0/1230752839 learned_addr learned my addr 172.21.15.203:0/1230752839 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-04T21:01:25.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.433+0000 7fd365ea8640 10 monclient(hunting): get_auth_request con 0x7fd360142dc0 auth_method 0 2024-04-04T21:01:25.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.433+0000 7fd365ea8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:25.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.433+0000 7fd365ea8640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.433+0000 7fd365ea8640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.433+0000 7fd364ea6640 10 monclient(hunting): get_auth_request con 0x7fd3600a9e30 auth_method 0 2024-04-04T21:01:25.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.433+0000 7fd364ea6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:25.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.433+0000 7fd364ea6640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.433+0000 7fd364ea6640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.433+0000 7fd365ea8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:25.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.433+0000 7fd365ea8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:25.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.433+0000 7fd365ea8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:25.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.434+0000 7fd364ea6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:25.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.434+0000 7fd364ea6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:25.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.434+0000 7fd364ea6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:25.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.435+0000 7fd365ea8640 10 monclient(hunting): handle_auth_done global_id 4155 payload 274 2024-04-04T21:01:25.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.435+0000 7fd365ea8640 1 -- 172.21.15.203:0/1230752839 >> v2:172.21.15.203:3300/0 conn(0x7fd3600a9e30 msgr2=0x7fd360142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:01:25.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.435+0000 7fd365ea8640 1 --2- 172.21.15.203:0/1230752839 >> v2:172.21.15.203:3300/0 conn(0x7fd3600a9e30 0x7fd360142880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.435+0000 7fd365ea8640 1 -- 172.21.15.203:0/1230752839 >> v1:172.21.15.203:6789/0 conn(0x7fd3600a9a50 legacy=0x7fd36013f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:01:25.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.435+0000 7fd365ea8640 10 monclient: _finish_hunting 0 2024-04-04T21:01:25.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.435+0000 7fd365ea8640 1 monclient: found mon.noname-a 2024-04-04T21:01:25.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.435+0000 7fd365ea8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:25.438 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.435+0000 7fd365ea8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:01:25.438 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.435+0000 7fd365ea8640 1 -- 172.21.15.203:0/1230752839 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3600aa250 con 0x7fd360142dc0 2024-04-04T21:01:25.438 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.437+0000 7fd365ea8640 1 --2- 172.21.15.203:0/1230752839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd360142dc0 0x7fd36014cef0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd36006c390 tx=0x7fd35802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a8a21f61b8cbc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:25.441 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd357fff640 1 -- 172.21.15.203:0/1230752839 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35802c9f0 con 0x7fd360142dc0 2024-04-04T21:01:25.441 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd357fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd357fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd357fff640 10 monclient: dump: 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd357fff640 10 monclient: _finish_auth 0 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd357fff640 10 monclient: _check_auth_tickets 2024-04-04T21:01:25.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd357fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd357fff640 1 -- 172.21.15.203:0/1230752839 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35802cb90 con 0x7fd360142dc0 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd357fff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd357fff640 1 -- 172.21.15.203:0/1230752839 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3580343f0 con 0x7fd360142dc0 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd36790b640 5 monclient: authenticate success, global_id 4155 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd36790b640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd36790b640 4 set_mon_vals no callback set 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd357fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd357fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.441+0000 7fd357fff640 10 monclient: dump: 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:25.443 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:25.444 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:25.444 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:25.444 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:25.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.442+0000 7fd36790b640 10 monclient: shutdown 2024-04-04T21:01:25.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.442+0000 7fd36790b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:25.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.442+0000 7fd36790b640 1 -- 172.21.15.203:0/1230752839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd360142dc0 msgr2=0x7fd36014cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:25.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.442+0000 7fd36790b640 1 --2- 172.21.15.203:0/1230752839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd360142dc0 0x7fd36014cef0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd36006c390 tx=0x7fd35802be50 comp rx=0 tx=0).stop 2024-04-04T21:01:25.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.442+0000 7fd36790b640 1 -- 172.21.15.203:0/1230752839 shutdown_connections 2024-04-04T21:01:25.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.442+0000 7fd36790b640 1 --2- 172.21.15.203:0/1230752839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd360142dc0 0x7fd36014cef0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.442+0000 7fd36790b640 1 --2- 172.21.15.203:0/1230752839 >> v2:172.21.15.203:3300/0 conn(0x7fd3600a9e30 0x7fd360142880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.442+0000 7fd36790b640 1 -- 172.21.15.203:0/1230752839 >> 172.21.15.203:0/1230752839 conn(0x7fd3600b8030 msgr2=0x7fd3600b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:25.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.442+0000 7fd36790b640 1 -- 172.21.15.203:0/1230752839 shutdown_connections 2024-04-04T21:01:25.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.442+0000 7fd36790b640 1 -- 172.21.15.203:0/1230752839 wait complete. 2024-04-04T21:01:25.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.443+0000 7fd36790b640 10 monclient: build_initial_monmap 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.443+0000 7fd36790b640 10 monclient: monmap: 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.443+0000 7fd36790b640 1 librados: starting msgr at 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.443+0000 7fd36790b640 1 librados: starting objecter 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.443+0000 7fd36790b640 1 Processor -- start 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.443+0000 7fd36790b640 1 -- start start 2024-04-04T21:01:25.448 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.443+0000 7fd36790b640 1 librados: setting wanted keys 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.443+0000 7fd36790b640 1 librados: calling monclient init 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.443+0000 7fd36790b640 10 monclient: init 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3600a9a50 0x7fd3601db360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 10 monclient(hunting): picked mon.noname-a con 0x7fd3600a9a50 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3600a9e30 0x7fd36013b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 10 monclient(hunting): picked mon.noname-b con 0x7fd3600a9e30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd360142dc0 0x7fd36013be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 10 monclient(hunting): picked mon.noname-c con 0x7fd360142dc0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3601433b0 con 0x7fd3600a9a50 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd360143230 con 0x7fd3600a9e30 2024-04-04T21:01:25.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36014d430 con 0x7fd360142dc0 2024-04-04T21:01:25.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:25.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd36790b640 10 monclient(hunting): authenticate will time out at 1325.594356s 2024-04-04T21:01:25.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd364ea6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3600a9e30 0x7fd36013b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:25.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.444+0000 7fd3656a7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3600a9a50 0x7fd3601db360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:25.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd365ea8640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd360142dc0 0x7fd36013be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:25.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd364ea6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3600a9e30 0x7fd36013b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:44736/0 (socket says 172.21.15.203:44736) 2024-04-04T21:01:25.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd364ea6640 1 -- 172.21.15.203:0/307924060 learned_addr learned my addr 172.21.15.203:0/307924060 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-04T21:01:25.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd364ea6640 10 monclient(hunting): get_auth_request con 0x7fd3600a9e30 auth_method 0 2024-04-04T21:01:25.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd364ea6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:25.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd364ea6640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd364ea6640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd3656a7640 10 monclient(hunting): get_auth_request con 0x7fd3600a9a50 auth_method 0 2024-04-04T21:01:25.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd3656a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd3656a7640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd3656a7640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd365ea8640 10 monclient(hunting): get_auth_request con 0x7fd360142dc0 auth_method 0 2024-04-04T21:01:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd365ea8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd365ea8640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd365ea8640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd364ea6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd364ea6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd364ea6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd3656a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd3656a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.445+0000 7fd3656a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.446+0000 7fd365ea8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.446+0000 7fd365ea8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:25.452 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.446+0000 7fd365ea8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:25.452 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.448+0000 7fd364ea6640 10 monclient(hunting): handle_auth_done global_id 4168 payload 824 2024-04-04T21:01:25.452 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.448+0000 7fd364ea6640 1 -- 172.21.15.203:0/307924060 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd360142dc0 msgr2=0x7fd36013be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:25.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.448+0000 7fd364ea6640 1 --2- 172.21.15.203:0/307924060 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd360142dc0 0x7fd36013be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.448+0000 7fd364ea6640 1 -- 172.21.15.203:0/307924060 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3600a9a50 msgr2=0x7fd3601db360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:25.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.448+0000 7fd364ea6640 1 --2- 172.21.15.203:0/307924060 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3600a9a50 0x7fd3601db360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.448+0000 7fd364ea6640 10 monclient: _finish_hunting 0 2024-04-04T21:01:25.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.448+0000 7fd364ea6640 1 monclient: found mon.noname-b 2024-04-04T21:01:25.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.448+0000 7fd364ea6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:25.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.448+0000 7fd364ea6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:01:25.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.448+0000 7fd364ea6640 1 -- 172.21.15.203:0/307924060 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3601dae90 con 0x7fd3600a9e30 2024-04-04T21:01:25.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.450+0000 7fd364ea6640 1 --2- 172.21.15.203:0/307924060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3600a9e30 0x7fd36013b930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00e9b0 tx=0x7fd35c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:25.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.450+0000 7fd355ffb640 10 client.?.objecter ms_handle_connect 0x7fd3600a9e30 2024-04-04T21:01:25.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.450+0000 7fd355ffb640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:25.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.453+0000 7fd355ffb640 1 -- 172.21.15.203:0/307924060 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c017070 con 0x7fd3600a9e30 2024-04-04T21:01:25.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.453+0000 7fd355ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:25.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.453+0000 7fd355ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:25.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.453+0000 7fd355ffb640 10 monclient: dump: 2024-04-04T21:01:25.455 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:25.455 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.455 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.455 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.455 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.453+0000 7fd355ffb640 10 monclient: _finish_auth 0 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd355ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd355ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd355ffb640 1 -- 172.21.15.203:0/307924060 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35c00c9a0 con 0x7fd3600a9e30 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd36790b640 5 monclient: authenticate success, global_id 4168 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd36790b640 10 monclient: _renew_subs 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd36790b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd36790b640 1 -- 172.21.15.203:0/307924060 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36013b250 con 0x7fd3600a9e30 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd355ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:25.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd355ffb640 1 -- 172.21.15.203:0/307924060 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c00cd90 con 0x7fd3600a9e30 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd355ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd355ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd355ffb640 10 monclient: dump: 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd3577fe640 4 set_mon_vals no callback set 2024-04-04T21:01:25.457 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd36790b640 10 client.4168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:25.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd36790b640 10 monclient: _renew_subs 2024-04-04T21:01:25.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd36790b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:25.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd36790b640 1 -- 172.21.15.203:0/307924060 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3601405c0 con 0x7fd3600a9e30 2024-04-04T21:01:25.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.454+0000 7fd36790b640 1 librados: init done 2024-04-04T21:01:25.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.455+0000 7fd36790b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:25.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.455+0000 7fd36790b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:25.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.455+0000 7fd36790b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:25.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.455+0000 7fd36790b640 1 -- 172.21.15.203:0/307924060 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd36004f050 con 0x7fd3600a9e30 2024-04-04T21:01:25.547 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:25.547+0000 7f6dbac4a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-04T21:01:25.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.557+0000 7fd355ffb640 1 -- 172.21.15.203:0/307924060 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd35c01a070 con 0x7fd3600a9e30 2024-04-04T21:01:25.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.557+0000 7fd355ffb640 1 -- 172.21.15.203:0/307924060 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fd35c0065d0 con 0x7fd3600a9e30 2024-04-04T21:01:25.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.557+0000 7fd355ffb640 10 client.4168.objecter ms_dispatch 0x7fd360000b90 osd_map(9..9 src has 1..9) v4 2024-04-04T21:01:25.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.557+0000 7fd355ffb640 3 client.4168.objecter handle_osd_map got epochs [9,9] > 0 2024-04-04T21:01:25.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.557+0000 7fd355ffb640 3 client.4168.objecter handle_osd_map decoding full epoch 9 2024-04-04T21:01:25.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.557+0000 7fd355ffb640 20 client.4168.objecter dump_active .. 0 homeless 2024-04-04T21:01:25.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.557+0000 7fd355ffb640 1 -- 172.21.15.203:0/307924060 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd35c006820 con 0x7fd3600a9e30 2024-04-04T21:01:25.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.557+0000 7fd355ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:25.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.557+0000 7fd355ffb640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:25.598 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:25.598+0000 7f05a03b8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-04T21:01:25.599 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:25.598+0000 7f6dbac4a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-04T21:01:25.630 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.629+0000 7fd36790b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "53aafc58-6463-4aeb-8395-e14fcfcee5de", "id": 7}] 2024-04-04T21:01:25.630 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.629+0000 7fd36790b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "53aafc58-6463-4aeb-8395-e14fcfcee5de", "id": 7}] 2024-04-04T21:01:25.630 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.629+0000 7fd36790b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:25.630 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.629+0000 7fd36790b640 1 -- 172.21.15.203:0/307924060 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "53aafc58-6463-4aeb-8395-e14fcfcee5de", "id": 7} v 0) v1 -- 0x7fd36013cfe0 con 0x7fd3600a9e30 2024-04-04T21:01:25.653 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:25.653+0000 7f05a03b8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-04T21:01:25.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.700+0000 7fd355ffb640 1 -- 172.21.15.203:0/307924060 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "53aafc58-6463-4aeb-8395-e14fcfcee5de", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd35c005ac0 con 0x7fd3600a9e30 2024-04-04T21:01:25.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.700+0000 7fd355ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "53aafc58-6463-4aeb-8395-e14fcfcee5de", "id": 7}] 2024-04-04T21:01:25.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.700+0000 7fd355ffb640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:01:25.701 INFO:teuthology.orchestra.run.smithi203.stdout:7 2024-04-04T21:01:25.703 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 10 librados: watch_flush enter 2024-04-04T21:01:25.703 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 10 librados: watch_flush exit 2024-04-04T21:01:25.703 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 20 client.4168.objecter shutdown clearing up homeless session... 2024-04-04T21:01:25.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 10 client.4168.objecter successfully canceled tick 2024-04-04T21:01:25.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 10 monclient: shutdown 2024-04-04T21:01:25.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:25.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 1 -- 172.21.15.203:0/307924060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3600a9e30 msgr2=0x7fd36013b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:25.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 1 --2- 172.21.15.203:0/307924060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3600a9e30 0x7fd36013b930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00e9b0 tx=0x7fd35c00ee70 comp rx=0 tx=0).stop 2024-04-04T21:01:25.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 1 -- 172.21.15.203:0/307924060 shutdown_connections 2024-04-04T21:01:25.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 1 --2- 172.21.15.203:0/307924060 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd360142dc0 0x7fd36013be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 1 --2- 172.21.15.203:0/307924060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3600a9e30 0x7fd36013b930 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 1 --2- 172.21.15.203:0/307924060 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3600a9a50 0x7fd3601db360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 1 -- 172.21.15.203:0/307924060 >> 172.21.15.203:0/307924060 conn(0x7fd3600b8030 msgr2=0x7fd36013d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:25.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 1 -- 172.21.15.203:0/307924060 shutdown_connections 2024-04-04T21:01:25.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 1 -- 172.21.15.203:0/307924060 wait complete. 2024-04-04T21:01:25.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-04T21:01:25.701+0000 7fd36790b640 1 librados: shutdown 2024-04-04T21:01:25.712 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-04T21:01:25.712 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-04T21:01:25.712 DEBUG:teuthology.orchestra.run.smithi106:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-04-04T21:01:25.715 INFO:tasks.ceph.osd.0:Started 2024-04-04T21:01:25.715 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-04T21:01:25.715 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-04T21:01:25.715 DEBUG:teuthology.orchestra.run.smithi106:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-04-04T21:01:25.717 INFO:tasks.ceph.osd.1:Started 2024-04-04T21:01:25.717 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-04T21:01:25.717 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-04T21:01:25.717 DEBUG:teuthology.orchestra.run.smithi106:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-04-04T21:01:25.719 INFO:tasks.ceph.osd.2:Started 2024-04-04T21:01:25.719 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-04T21:01:25.720 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-04T21:01:25.720 DEBUG:teuthology.orchestra.run.smithi106:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-04-04T21:01:25.721 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:25.720+0000 7f6dbac4a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-04T21:01:25.722 INFO:tasks.ceph.osd.3:Started 2024-04-04T21:01:25.722 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-04T21:01:25.722 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-04T21:01:25.722 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-04-04T21:01:25.724 INFO:tasks.ceph.osd.4:Started 2024-04-04T21:01:25.724 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-04T21:01:25.725 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-04T21:01:25.725 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-04-04T21:01:25.727 INFO:tasks.ceph.osd.5:Started 2024-04-04T21:01:25.727 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-04T21:01:25.727 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-04T21:01:25.727 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-04-04T21:01:25.729 INFO:tasks.ceph.osd.6:Started 2024-04-04T21:01:25.730 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-04T21:01:25.730 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-04T21:01:25.730 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-04-04T21:01:25.732 INFO:tasks.ceph.osd.7:Started 2024-04-04T21:01:25.733 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-04T21:01:25.809 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-04T21:01:25.809+0000 7f05a03b8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-04T21:01:25.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.871+0000 7f80e1b2f640 10 monclient: get_monmap_and_config 2024-04-04T21:01:25.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.871+0000 7f80e1b2f640 10 monclient: build_initial_monmap 2024-04-04T21:01:25.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.871+0000 7f80e1b2f640 10 monclient: monmap: 2024-04-04T21:01:25.872 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:01:25.872 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.872 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:01:25.872234+0000 2024-04-04T21:01:25.872 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:01:25.872234+0000 2024-04-04T21:01:25.872 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:25.872 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:25.872 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:25.872 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:25.872 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:25.873 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:25.873 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:25.873 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:01:25.876 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.872+0000 7f80e1b2f640 1 Processor -- start 2024-04-04T21:01:25.876 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.872+0000 7f80e1b2f640 1 -- start start 2024-04-04T21:01:25.876 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.872+0000 7f80e1b2f640 10 monclient: init 2024-04-04T21:01:25.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:25.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-04T21:01:25.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f80dc0a9a50 0x7f80dc0a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 10 monclient(hunting): picked mon.noname-a con 0x7f80dc0a9a50 addr v2:172.21.15.106:3300/0 2024-04-04T21:01:25.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f80dc13f6a0 0x7f80dc13fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 10 monclient(hunting): picked mon.noname-c con 0x7f80dc13f6a0 addr v2:172.21.15.203:3300/0 2024-04-04T21:01:25.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80dc142880 0x7f80dc142d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 10 monclient(hunting): picked mon.noname-b con 0x7f80dc142880 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:25.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80dc0aa370 con 0x7f80dc0a9a50 2024-04-04T21:01:25.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80dc0ab750 con 0x7f80dc13f6a0 2024-04-04T21:01:25.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80dc140120 con 0x7f80dc142880 2024-04-04T21:01:25.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:25.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.873+0000 7f80e1b2f640 10 monclient(hunting): authenticate will time out at 1335.140422s 2024-04-04T21:01:25.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.874+0000 7f80daffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f80dc13f6a0 0x7f80dc13fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:25.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.874+0000 7f80dbfff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80dc142880 0x7f80dc142d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:25.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.874+0000 7f80daffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f80dc13f6a0 0x7f80dc13fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:55790/0 (socket says 172.21.15.106:55790) 2024-04-04T21:01:25.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.874+0000 7f80daffd640 1 -- 172.21.15.106:0/2163839837 learned_addr learned my addr 172.21.15.106:0/2163839837 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:01:25.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.874+0000 7f80daffd640 10 monclient(hunting): get_auth_request con 0x7f80dc13f6a0 auth_method 0 2024-04-04T21:01:25.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.874+0000 7f80daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:25.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.874+0000 7f80daffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.874+0000 7f80daffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.875+0000 7f80daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:25.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.875+0000 7f80daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:25.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.875+0000 7f80daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:25.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.876+0000 7f80dbfff640 10 monclient(hunting): get_auth_request con 0x7f80dc142880 auth_method 0 2024-04-04T21:01:25.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.876+0000 7f80dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:25.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.876+0000 7f80dbfff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.876+0000 7f80dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.876+0000 7f80dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:25.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.876+0000 7f80dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:25.881 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.876+0000 7f80dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:25.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.878+0000 7f80daffd640 10 monclient(hunting): handle_auth_done global_id 4171 payload 274 2024-04-04T21:01:25.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.878+0000 7f80daffd640 1 -- 172.21.15.106:0/2163839837 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80dc142880 msgr2=0x7f80dc142d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:01:25.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.878+0000 7f80daffd640 1 --2- 172.21.15.106:0/2163839837 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80dc142880 0x7f80dc142d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.878+0000 7f80daffd640 1 -- 172.21.15.106:0/2163839837 >> v2:172.21.15.106:3300/0 conn(0x7f80dc0a9a50 msgr2=0x7f80dc0a9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:01:25.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.878+0000 7f80daffd640 1 --2- 172.21.15.106:0/2163839837 >> v2:172.21.15.106:3300/0 conn(0x7f80dc0a9a50 0x7f80dc0a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.878+0000 7f80daffd640 10 monclient: _finish_hunting 0 2024-04-04T21:01:25.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.878+0000 7f80daffd640 1 monclient: found mon.noname-c 2024-04-04T21:01:25.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.878+0000 7f80daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:25.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.878+0000 7f80daffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:01:25.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.878+0000 7f80daffd640 1 -- 172.21.15.106:0/2163839837 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80dc0aa660 con 0x7f80dc13f6a0 2024-04-04T21:01:25.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.880+0000 7f80daffd640 1 --2- 172.21.15.106:0/2163839837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80dc13f6a0 0x7f80dc13fac0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f80dc06c390 tx=0x7f80d002b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5301ad11da0fa63 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:25.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.886+0000 7f80da7fc640 1 -- 172.21.15.106:0/2163839837 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d00317f0 con 0x7f80dc13f6a0 2024-04-04T21:01:25.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.886+0000 7f80da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:25.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.886+0000 7f80da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:01:25.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.886+0000 7f80da7fc640 10 monclient: dump: 2024-04-04T21:01:25.890 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:01:25.890 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.890 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.890 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.890 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:25.890 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:25.890 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:25.890 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:25.890 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:25.890 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:01:25.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.886+0000 7f80da7fc640 10 monclient: _finish_auth 0 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.886+0000 7f80da7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.886+0000 7f80da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.886+0000 7f80da7fc640 1 -- 172.21.15.106:0/2163839837 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d0037070 con 0x7f80dc13f6a0 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.886+0000 7f80da7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.886+0000 7f80da7fc640 1 -- 172.21.15.106:0/2163839837 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d0031bb0 con 0x7f80dc13f6a0 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.886+0000 7f80e1b2f640 5 monclient: authenticate success, global_id 4171 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.886+0000 7f80e1b2f640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.886+0000 7f80e1b2f640 4 set_mon_vals no callback set 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.887+0000 7f80da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.887+0000 7f80da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.887+0000 7f80da7fc640 10 monclient: dump: 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.891 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.887+0000 7f80e1b2f640 10 monclient: shutdown 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.887+0000 7f80e1b2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.887+0000 7f80e1b2f640 1 -- 172.21.15.106:0/2163839837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80dc13f6a0 msgr2=0x7f80dc13fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.887+0000 7f80e1b2f640 1 --2- 172.21.15.106:0/2163839837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80dc13f6a0 0x7f80dc13fac0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f80dc06c390 tx=0x7f80d002b890 comp rx=0 tx=0).stop 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.887+0000 7f80e1b2f640 1 -- 172.21.15.106:0/2163839837 shutdown_connections 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.887+0000 7f80e1b2f640 1 --2- 172.21.15.106:0/2163839837 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80dc142880 0x7f80dc142d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.887+0000 7f80e1b2f640 1 --2- 172.21.15.106:0/2163839837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80dc13f6a0 0x7f80dc13fac0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.887+0000 7f80e1b2f640 1 --2- 172.21.15.106:0/2163839837 >> v2:172.21.15.106:3300/0 conn(0x7f80dc0a9a50 0x7f80dc0a9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.887+0000 7f80e1b2f640 1 -- 172.21.15.106:0/2163839837 >> 172.21.15.106:0/2163839837 conn(0x7f80dc0b8030 msgr2=0x7f80dc0b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.887+0000 7f80e1b2f640 1 -- 172.21.15.106:0/2163839837 shutdown_connections 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.887+0000 7f80e1b2f640 1 -- 172.21.15.106:0/2163839837 wait complete. 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.888+0000 7f80e1b2f640 10 monclient: build_initial_monmap 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.888+0000 7f80e1b2f640 10 monclient: monmap: 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.888+0000 7f80e1b2f640 1 librados: starting msgr at 2024-04-04T21:01:25.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.888+0000 7f80e1b2f640 1 librados: starting objecter 2024-04-04T21:01:25.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.889+0000 7f80e1b2f640 1 Processor -- start 2024-04-04T21:01:25.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.889+0000 7f80e1b2f640 1 -- start start 2024-04-04T21:01:25.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.889+0000 7f80e1b2f640 1 librados: setting wanted keys 2024-04-04T21:01:25.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.889+0000 7f80e1b2f640 1 librados: calling monclient init 2024-04-04T21:01:25.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.889+0000 7f80e1b2f640 10 monclient: init 2024-04-04T21:01:25.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:25.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-04T21:01:25.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80dc0a9a50 0x7f80dc145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 10 monclient(hunting): picked mon.noname-b con 0x7f80dc0a9a50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:25.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80dc13f6a0 0x7f80dc1df400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 10 monclient(hunting): picked mon.noname-a con 0x7f80dc13f6a0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:25.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80dc142880 0x7f80dc1dbf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:25.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 10 monclient(hunting): picked mon.noname-c con 0x7f80dc142880 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:25.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80dc145b60 con 0x7f80dc13f6a0 2024-04-04T21:01:25.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80dc143460 con 0x7f80dc0a9a50 2024-04-04T21:01:25.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:25.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80dc145ce0 con 0x7f80dc142880 2024-04-04T21:01:25.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:25.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80e1b2f640 10 monclient(hunting): authenticate will time out at 1335.157869s 2024-04-04T21:01:25.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.890+0000 7f80db7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80dc0a9a50 0x7f80dc145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:25.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.891+0000 7f80db7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80dc0a9a50 0x7f80dc145160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:55792/0 (socket says 172.21.15.106:55792) 2024-04-04T21:01:25.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.891+0000 7f80db7fe640 1 -- 172.21.15.106:0/1987702120 learned_addr learned my addr 172.21.15.106:0/1987702120 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.891+0000 7f80db7fe640 10 monclient(hunting): get_auth_request con 0x7f80dc0a9a50 auth_method 0 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.891+0000 7f80db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.891+0000 7f80db7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.891+0000 7f80db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.891+0000 7f80dbfff640 1 --2- 172.21.15.106:0/1987702120 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80dc142880 0x7f80dc1dbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.891+0000 7f80dbfff640 10 monclient(hunting): get_auth_request con 0x7f80dc142880 auth_method 0 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.891+0000 7f80dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.891+0000 7f80dbfff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.891+0000 7f80dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.892+0000 7f80db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.892+0000 7f80db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.892+0000 7f80db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.892+0000 7f80dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.892+0000 7f80dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.892+0000 7f80dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:25.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.896+0000 7f80dbfff640 10 monclient(hunting): handle_auth_done global_id 4139 payload 824 2024-04-04T21:01:25.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.896+0000 7f80dbfff640 1 -- 172.21.15.106:0/1987702120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80dc0a9a50 msgr2=0x7f80dc145160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:25.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.896+0000 7f80dbfff640 1 --2- 172.21.15.106:0/1987702120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80dc0a9a50 0x7f80dc145160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.896+0000 7f80dbfff640 1 -- 172.21.15.106:0/1987702120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80dc13f6a0 msgr2=0x7f80dc1df400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:25.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.896+0000 7f80dbfff640 1 --2- 172.21.15.106:0/1987702120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80dc13f6a0 0x7f80dc1df400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:25.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.896+0000 7f80dbfff640 10 monclient: _finish_hunting 0 2024-04-04T21:01:25.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.896+0000 7f80dbfff640 1 monclient: found mon.noname-c 2024-04-04T21:01:25.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.896+0000 7f80dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:25.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.896+0000 7f80dbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:01:25.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.896+0000 7f80dbfff640 1 -- 172.21.15.106:0/1987702120 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80dc1df940 con 0x7f80dc142880 2024-04-04T21:01:25.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.899+0000 7f80dbfff640 1 --2- 172.21.15.106:0/1987702120 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80dc142880 0x7f80dc1dbf00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f80d400eba0 tx=0x7f80d400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:25.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.899+0000 7f80c3fff640 10 client.?.objecter ms_handle_connect 0x7f80dc142880 2024-04-04T21:01:25.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.899+0000 7f80c3fff640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:25.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.904+0000 7f80c3fff640 1 -- 172.21.15.106:0/1987702120 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d4018020 con 0x7f80dc142880 2024-04-04T21:01:25.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.904+0000 7f80c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:25.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.904+0000 7f80c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:01:25.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.904+0000 7f80c3fff640 10 monclient: dump: 2024-04-04T21:01:25.909 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:01:25.909 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.909 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.909 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.909 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:25.909 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:25.909 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:25.909 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:25.909 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:25.909 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:01:25.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.904+0000 7f80c3fff640 10 monclient: _finish_auth 0 2024-04-04T21:01:25.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.904+0000 7f80c3fff640 10 monclient: _check_auth_tickets 2024-04-04T21:01:25.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.904+0000 7f80c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:25.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80c3fff640 1 -- 172.21.15.106:0/1987702120 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d400cd20 con 0x7f80dc142880 2024-04-04T21:01:25.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80c3fff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:25.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80c3fff640 1 -- 172.21.15.106:0/1987702120 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d40146c0 con 0x7f80dc142880 2024-04-04T21:01:25.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:25.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:01:25.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80c3fff640 10 monclient: dump: 2024-04-04T21:01:25.910 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:01:25.910 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:25.910 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.910 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:25.910 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:25.911 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:25.911 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:25.911 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:25.911 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:25.911 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:01:25.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80e1b2f640 5 monclient: authenticate success, global_id 4139 2024-04-04T21:01:25.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80e1b2f640 10 monclient: _renew_subs 2024-04-04T21:01:25.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80e1b2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:25.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80e1b2f640 1 -- 172.21.15.106:0/1987702120 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80dc1dfbf0 con 0x7f80dc142880 2024-04-04T21:01:25.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80e1b2f640 10 client.4139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:25.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80e1b2f640 10 monclient: _renew_subs 2024-04-04T21:01:25.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80e1b2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:25.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80e1b2f640 1 -- 172.21.15.106:0/1987702120 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80dc1e0110 con 0x7f80dc142880 2024-04-04T21:01:25.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80e1b2f640 1 librados: init done 2024-04-04T21:01:25.912 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.905+0000 7f80d9ffb640 4 set_mon_vals no callback set 2024-04-04T21:01:25.912 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.906+0000 7f80e1b2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:25.912 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.906+0000 7f80e1b2f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:25.912 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.906+0000 7f80e1b2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:25.912 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.906+0000 7f80e1b2f640 1 -- 172.21.15.106:0/1987702120 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80bc005080 con 0x7f80dc142880 2024-04-04T21:01:25.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.911+0000 7f80c3fff640 1 -- 172.21.15.106:0/1987702120 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f80d4005d10 con 0x7f80dc142880 2024-04-04T21:01:25.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.911+0000 7f80c3fff640 1 -- 172.21.15.106:0/1987702120 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f80d401dd00 con 0x7f80dc142880 2024-04-04T21:01:25.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.911+0000 7f80c3fff640 10 client.4139.objecter ms_dispatch 0x7f80dc000b90 osd_map(10..10 src has 1..10) v4 2024-04-04T21:01:25.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.911+0000 7f80c3fff640 3 client.4139.objecter handle_osd_map got epochs [10,10] > 0 2024-04-04T21:01:25.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.911+0000 7f80c3fff640 3 client.4139.objecter handle_osd_map decoding full epoch 10 2024-04-04T21:01:25.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:25.911+0000 7f80c3fff640 20 client.4139.objecter dump_active .. 0 homeless 2024-04-04T21:01:26.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.093+0000 7f80c3fff640 1 -- 172.21.15.106:0/1987702120 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f80d4022030 con 0x7f80dc142880 2024-04-04T21:01:26.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.093+0000 7f80c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:26.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.093+0000 7f80c3fff640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:26.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.217+0000 7f80e1b2f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:26.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.217+0000 7f80e1b2f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:26.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.217+0000 7f80e1b2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:26.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.217+0000 7f80e1b2f640 1 -- 172.21.15.106:0/1987702120 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80bc005640 con 0x7f80dc142880 2024-04-04T21:01:26.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.377+0000 7f80c3fff640 1 -- 172.21.15.106:0/1987702120 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mgrmap(e 2) v1 ==== 44914+0+0 (secure 0 0 0) 0x7f80d401b020 con 0x7f80dc142880 2024-04-04T21:01:26.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.454+0000 7f80c3fff640 1 -- 172.21.15.106:0/1987702120 <== mon.2 v2:172.21.15.106:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f80d4010040 con 0x7f80dc142880 2024-04-04T21:01:26.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.454+0000 7f80c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:26.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.454+0000 7f80c3fff640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:01:26.455 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:01:26.455 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":10,"fsid":"8f89ce43-1bcb-41cf-9555-c40eeb258155","created":"2024-04-04T21:01:20.057083+0000","modified":"2024-04-04T21:01:25.642687+0000","last_up_change":"0.000000","last_in_change":"2024-04-04T21:01:25.642687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"28337134-5253-454a-bcda-47267b05b2c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ae336a0e-2a90-4323-9f0f-1701fffe6b66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"773087e0-78c9-4d14-bf0a-fb605c21f92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07da9513-c746-46d1-995f-6f8e91338b84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c47d034c-1db6-4332-bf24-722fb2846808","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"588caa62-e773-44ae-94e5-ece54fe980d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ef776e12-7c80-4095-b173-5b2d24996f95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"53aafc58-6463-4aeb-8395-e14fcfcee5de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-04T21:01:26.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.455+0000 7f80e1b2f640 10 librados: watch_flush enter 2024-04-04T21:01:26.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.455+0000 7f80e1b2f640 10 librados: watch_flush exit 2024-04-04T21:01:26.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.455+0000 7f80e1b2f640 20 client.4139.objecter shutdown clearing up homeless session... 2024-04-04T21:01:26.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.455+0000 7f80e1b2f640 10 client.4139.objecter successfully canceled tick 2024-04-04T21:01:26.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.455+0000 7f80e1b2f640 10 monclient: shutdown 2024-04-04T21:01:26.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.455+0000 7f80e1b2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:26.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.455+0000 7f80e1b2f640 1 -- 172.21.15.106:0/1987702120 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80dc142880 msgr2=0x7f80dc1dbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:26.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.455+0000 7f80e1b2f640 1 --2- 172.21.15.106:0/1987702120 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80dc142880 0x7f80dc1dbf00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f80d400eba0 tx=0x7f80d400c490 comp rx=0 tx=0).stop 2024-04-04T21:01:26.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.455+0000 7f80e1b2f640 1 -- 172.21.15.106:0/1987702120 shutdown_connections 2024-04-04T21:01:26.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.455+0000 7f80e1b2f640 1 --2- 172.21.15.106:0/1987702120 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80dc142880 0x7f80dc1dbf00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:26.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.455+0000 7f80e1b2f640 1 --2- 172.21.15.106:0/1987702120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80dc13f6a0 0x7f80dc1df400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:26.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.455+0000 7f80e1b2f640 1 --2- 172.21.15.106:0/1987702120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80dc0a9a50 0x7f80dc145160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:26.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.455+0000 7f80e1b2f640 1 -- 172.21.15.106:0/1987702120 >> 172.21.15.106:0/1987702120 conn(0x7f80dc0b8030 msgr2=0x7f80dc0b64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:26.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.456+0000 7f80e1b2f640 1 -- 172.21.15.106:0/1987702120 shutdown_connections 2024-04-04T21:01:26.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.456+0000 7f80e1b2f640 1 -- 172.21.15.106:0/1987702120 wait complete. 2024-04-04T21:01:26.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:01:26.456+0000 7f80e1b2f640 1 librados: shutdown 2024-04-04T21:01:26.470 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-04T21:01:26.471 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-04T21:01:27.277 DEBUG:teuthology.orchestra.run.smithi106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-04T21:01:27.421 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.420+0000 7fa1515dd640 10 monclient: get_monmap_and_config 2024-04-04T21:01:27.421 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.420+0000 7fa1515dd640 10 monclient: build_initial_monmap 2024-04-04T21:01:27.421 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.420+0000 7fa1515dd640 10 monclient: monmap: 2024-04-04T21:01:27.421 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:01:27.421 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:27.422 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:01:27.421503+0000 2024-04-04T21:01:27.422 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:01:27.421503+0000 2024-04-04T21:01:27.422 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:27.422 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:27.422 INFO:teuthology.misc.health.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:27.422 INFO:teuthology.misc.health.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:27.422 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:27.422 INFO:teuthology.misc.health.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:27.422 INFO:teuthology.misc.health.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:27.422 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:27.424 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.421+0000 7fa1515dd640 1 Processor -- start 2024-04-04T21:01:27.424 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.421+0000 7fa1515dd640 1 -- start start 2024-04-04T21:01:27.425 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.421+0000 7fa1515dd640 10 monclient: init 2024-04-04T21:01:27.425 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.422+0000 7fa1515dd640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:27.425 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.422+0000 7fa1515dd640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-04-04T21:01:27.425 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.422+0000 7fa1515dd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa14c0b87e0 0x7fa14c142650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:27.425 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.422+0000 7fa1515dd640 10 monclient(hunting): picked mon.noname-c con 0x7fa14c0b87e0 addr v2:172.21.15.203:3300/0 2024-04-04T21:01:27.425 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.422+0000 7fa1515dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa14c0aa300 addr v1:172.21.15.106:6789/0 2024-04-04T21:01:27.425 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.422+0000 7fa1515dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa14c13fc70 addr v1:172.21.15.203:6789/0 2024-04-04T21:01:27.425 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.422+0000 7fa1515dd640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c0ab360 con 0x7fa14c0aa300 2024-04-04T21:01:27.425 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.422+0000 7fa1515dd640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c0ab640 con 0x7fa14c13fc70 2024-04-04T21:01:27.426 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.422+0000 7fa1515dd640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:27.426 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.422+0000 7fa1515dd640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa14c0b8bc0 con 0x7fa14c0b87e0 2024-04-04T21:01:27.426 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.422+0000 7fa1515dd640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:27.426 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.422+0000 7fa1515dd640 10 monclient(hunting): authenticate will time out at 1336.689743s 2024-04-04T21:01:27.427 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.425+0000 7fa14b7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa14c13fc70 0x7fa14c14cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:52398/0 (socket says 172.21.15.106:52398) 2024-04-04T21:01:27.427 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.425+0000 7fa14b7fe640 1 -- 172.21.15.106:0/3160176124 learned_addr learned my addr 172.21.15.106:0/3160176124 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:01:27.430 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.430+0000 7fa14affd640 1 --2- 172.21.15.106:0/3160176124 >> v2:172.21.15.203:3300/0 conn(0x7fa14c0b87e0 0x7fa14c142650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:27.432 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.431+0000 7fa14affd640 10 monclient(hunting): get_auth_request con 0x7fa14c0b87e0 auth_method 0 2024-04-04T21:01:27.432 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.431+0000 7fa14affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:27.432 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.431+0000 7fa14affd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:27.432 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.431+0000 7fa14affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:27.433 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.432+0000 7fa14affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:27.433 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.432+0000 7fa14affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:27.433 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.432+0000 7fa14affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:27.437 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.436+0000 7fa14affd640 10 monclient(hunting): handle_auth_done global_id 4177 payload 274 2024-04-04T21:01:27.437 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.436+0000 7fa14affd640 1 -- 172.21.15.106:0/3160176124 >> v1:172.21.15.203:6789/0 conn(0x7fa14c13fc70 legacy=0x7fa14c14cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:27.437 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.436+0000 7fa14affd640 1 -- 172.21.15.106:0/3160176124 >> v1:172.21.15.106:6789/0 conn(0x7fa14c0aa300 legacy=0x7fa14c13f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:01:27.437 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.436+0000 7fa14affd640 10 monclient: _finish_hunting 0 2024-04-04T21:01:27.437 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.436+0000 7fa14affd640 1 monclient: found mon.noname-c 2024-04-04T21:01:27.437 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.436+0000 7fa14affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:27.437 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.436+0000 7fa14affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:01:27.437 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.436+0000 7fa14affd640 1 -- 172.21.15.106:0/3160176124 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c140190 con 0x7fa14c0b87e0 2024-04-04T21:01:27.441 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.441+0000 7fa14affd640 1 --2- 172.21.15.106:0/3160176124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa14c0b87e0 0x7fa14c142650 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0097a0 tx=0x7fa13c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94153daa2cb43494 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:27.444 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.444+0000 7fa14a7fc640 1 -- 172.21.15.106:0/3160176124 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c0315a0 con 0x7fa14c0b87e0 2024-04-04T21:01:27.445 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.444+0000 7fa14a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:27.445 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.444+0000 7fa14a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:01:27.445 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.444+0000 7fa14a7fc640 10 monclient: dump: 2024-04-04T21:01:27.445 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:27.445 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:27.445 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:27.445 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:27.445 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:27.445 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:27.445 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:27.445 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:27.445 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:27.445 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:27.445 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.444+0000 7fa14a7fc640 10 monclient: _finish_auth 0 2024-04-04T21:01:27.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.444+0000 7fa14a7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:01:27.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.444+0000 7fa14a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:27.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.444+0000 7fa1515dd640 5 monclient: authenticate success, global_id 4177 2024-04-04T21:01:27.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.444+0000 7fa1515dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:01:27.449 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.446+0000 7fa14a7fc640 1 -- 172.21.15.106:0/3160176124 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13c037070 con 0x7fa14c0b87e0 2024-04-04T21:01:27.449 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.446+0000 7fa14a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:27.449 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.446+0000 7fa14a7fc640 1 -- 172.21.15.106:0/3160176124 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c031960 con 0x7fa14c0b87e0 2024-04-04T21:01:27.449 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.446+0000 7fa14a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:27.449 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.446+0000 7fa14a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:27.449 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.446+0000 7fa14a7fc640 10 monclient: dump: 2024-04-04T21:01:27.449 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:27.449 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:27.449 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:27.449 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:27.449 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:27.449 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:27.449 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:27.450 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:27.450 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:27.450 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:27.450 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.446+0000 7fa1515dd640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:27.450 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.446+0000 7fa1515dd640 4 set_mon_vals no callback set 2024-04-04T21:01:27.450 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.447+0000 7fa1515dd640 10 monclient: shutdown 2024-04-04T21:01:27.450 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.447+0000 7fa1515dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:27.450 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.447+0000 7fa1515dd640 1 -- 172.21.15.106:0/3160176124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa14c0b87e0 msgr2=0x7fa14c142650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:27.450 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.447+0000 7fa1515dd640 1 --2- 172.21.15.106:0/3160176124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa14c0b87e0 0x7fa14c142650 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0097a0 tx=0x7fa13c02b840 comp rx=0 tx=0).stop 2024-04-04T21:01:27.450 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.447+0000 7fa1515dd640 1 -- 172.21.15.106:0/3160176124 shutdown_connections 2024-04-04T21:01:27.450 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.447+0000 7fa1515dd640 1 --2- 172.21.15.106:0/3160176124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa14c0b87e0 0x7fa14c142650 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:27.450 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.447+0000 7fa1515dd640 1 -- 172.21.15.106:0/3160176124 >> 172.21.15.106:0/3160176124 conn(0x7fa14c0727d0 msgr2=0x7fa14c072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:27.450 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.448+0000 7fa1515dd640 1 -- 172.21.15.106:0/3160176124 shutdown_connections 2024-04-04T21:01:27.451 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.448+0000 7fa1515dd640 1 -- 172.21.15.106:0/3160176124 wait complete. 2024-04-04T21:01:27.451 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.448+0000 7fa1515dd640 10 monclient: build_initial_monmap 2024-04-04T21:01:27.451 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.448+0000 7fa1515dd640 10 monclient: monmap: 2024-04-04T21:01:27.451 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:01:27.451 INFO:teuthology.misc.health.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:27.451 INFO:teuthology.misc.health.smithi106.stderr:last_changed 0.000000 2024-04-04T21:01:27.451 INFO:teuthology.misc.health.smithi106.stderr:created 0.000000 2024-04-04T21:01:27.451 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:27.451 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:27.451 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:27.451 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:27.451 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:27.451 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:27.454 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.449+0000 7fa1515dd640 1 librados: starting msgr at 2024-04-04T21:01:27.454 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.449+0000 7fa1515dd640 1 librados: starting objecter 2024-04-04T21:01:27.454 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.449+0000 7fa1515dd640 1 Processor -- start 2024-04-04T21:01:27.454 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.449+0000 7fa1515dd640 1 -- start start 2024-04-04T21:01:27.454 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.449+0000 7fa1515dd640 1 librados: setting wanted keys 2024-04-04T21:01:27.455 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.449+0000 7fa1515dd640 1 librados: calling monclient init 2024-04-04T21:01:27.455 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.449+0000 7fa1515dd640 10 monclient: init 2024-04-04T21:01:27.455 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.450+0000 7fa1515dd640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:27.455 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.450+0000 7fa1515dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:01:27.455 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.450+0000 7fa1515dd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa14c0aa300 0x7fa14c1465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:27.455 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.450+0000 7fa1515dd640 10 monclient(hunting): picked mon.noname-a con 0x7fa14c0aa300 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:27.455 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.450+0000 7fa1515dd640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa14c0b87e0 0x7fa14c1d82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:27.455 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.451+0000 7fa1515dd640 10 monclient(hunting): picked mon.noname-c con 0x7fa14c0b87e0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:27.455 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.451+0000 7fa1515dd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa14c13fc70 0x7fa14c1db5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:27.455 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.451+0000 7fa1515dd640 10 monclient(hunting): picked mon.noname-b con 0x7fa14c13fc70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:27.455 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.451+0000 7fa1515dd640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:27.455 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.451+0000 7fa1515dd640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c0a9b80 con 0x7fa14c0aa300 2024-04-04T21:01:27.455 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.451+0000 7fa1515dd640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:27.455 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.451+0000 7fa1515dd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c0a9a00 con 0x7fa14c13fc70 2024-04-04T21:01:27.456 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.451+0000 7fa1515dd640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:27.456 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.451+0000 7fa1515dd640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c142b90 con 0x7fa14c0b87e0 2024-04-04T21:01:27.456 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.451+0000 7fa1515dd640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:27.456 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.451+0000 7fa1515dd640 10 monclient(hunting): authenticate will time out at 1336.718859s 2024-04-04T21:01:27.456 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa14b7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa14c13fc70 0x7fa14c1db5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:27.456 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa143fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa14c0b87e0 0x7fa14c1d82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:27.456 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa143fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa14c0b87e0 0x7fa14c1d82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:53320/0 (socket says 172.21.15.106:53320) 2024-04-04T21:01:27.456 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa143fff640 1 -- 172.21.15.106:0/3939428494 learned_addr learned my addr 172.21.15.106:0/3939428494 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:01:27.456 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa143fff640 10 monclient(hunting): get_auth_request con 0x7fa14c0b87e0 auth_method 0 2024-04-04T21:01:27.456 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa143fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:27.456 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa143fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:27.456 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa143fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:27.456 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa14b7fe640 10 monclient(hunting): get_auth_request con 0x7fa14c13fc70 auth_method 0 2024-04-04T21:01:27.456 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa14b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:27.457 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa14b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:27.457 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa14b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:27.457 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa143fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:27.457 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa143fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:27.457 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.452+0000 7fa143fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:27.457 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.453+0000 7fa14b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:27.457 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.453+0000 7fa14b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:27.457 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.453+0000 7fa14b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:27.457 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.456+0000 7fa143fff640 10 monclient(hunting): handle_auth_done global_id 4157 payload 824 2024-04-04T21:01:27.458 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.456+0000 7fa143fff640 1 -- 172.21.15.106:0/3939428494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa14c13fc70 msgr2=0x7fa14c1db5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:27.458 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.456+0000 7fa143fff640 1 --2- 172.21.15.106:0/3939428494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa14c13fc70 0x7fa14c1db5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:27.458 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.456+0000 7fa143fff640 1 -- 172.21.15.106:0/3939428494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa14c0aa300 msgr2=0x7fa14c1465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:27.458 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.456+0000 7fa143fff640 1 --2- 172.21.15.106:0/3939428494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa14c0aa300 0x7fa14c1465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:27.458 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.456+0000 7fa143fff640 10 monclient: _finish_hunting 0 2024-04-04T21:01:27.458 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.456+0000 7fa143fff640 1 monclient: found mon.noname-c 2024-04-04T21:01:27.458 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.456+0000 7fa143fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:27.458 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.456+0000 7fa143fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:01:27.458 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.456+0000 7fa143fff640 1 -- 172.21.15.106:0/3939428494 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c1d8f80 con 0x7fa14c0b87e0 2024-04-04T21:01:27.460 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.460+0000 7fa143fff640 1 --2- 172.21.15.106:0/3939428494 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa14c0b87e0 0x7fa14c1d82f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa13400c410 tx=0x7fa13400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:27.460 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.460+0000 7fa1437fe640 10 client.?.objecter ms_handle_connect 0x7fa14c0b87e0 2024-04-04T21:01:27.460 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.460+0000 7fa1437fe640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:27.469 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.466+0000 7fa1437fe640 1 -- 172.21.15.106:0/3939428494 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa134017070 con 0x7fa14c0b87e0 2024-04-04T21:01:27.469 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.466+0000 7fa1437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:27.469 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:01:27.469 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1437fe640 10 monclient: dump: 2024-04-04T21:01:27.469 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:27.469 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:27.469 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:27.469 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:27.469 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:27.469 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:27.469 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:27.470 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:27.470 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:27.470 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:27.470 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1437fe640 10 monclient: _finish_auth 0 2024-04-04T21:01:27.470 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1437fe640 10 monclient: _check_auth_tickets 2024-04-04T21:01:27.470 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:27.470 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1437fe640 1 -- 172.21.15.106:0/3939428494 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1340035f0 con 0x7fa14c0b87e0 2024-04-04T21:01:27.470 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1515dd640 5 monclient: authenticate success, global_id 4157 2024-04-04T21:01:27.470 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1515dd640 10 monclient: _renew_subs 2024-04-04T21:01:27.470 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1515dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:27.470 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1515dd640 1 -- 172.21.15.106:0/3939428494 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14c1cf9c0 con 0x7fa14c0b87e0 2024-04-04T21:01:27.470 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1437fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:27.470 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1437fe640 1 -- 172.21.15.106:0/3939428494 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa134003a20 con 0x7fa14c0b87e0 2024-04-04T21:01:27.470 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1515dd640 10 client.4157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1437fe640 10 monclient: dump: 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1515dd640 10 monclient: _renew_subs 2024-04-04T21:01:27.471 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa149ffb640 4 set_mon_vals no callback set 2024-04-04T21:01:27.472 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1515dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:27.472 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1515dd640 1 -- 172.21.15.106:0/3939428494 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14c1dbce0 con 0x7fa14c0b87e0 2024-04-04T21:01:27.472 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.467+0000 7fa1515dd640 1 librados: init done 2024-04-04T21:01:27.472 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.468+0000 7fa1515dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:27.472 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.468+0000 7fa1515dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:27.472 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.468+0000 7fa1515dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:27.472 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.468+0000 7fa1515dd640 1 -- 172.21.15.106:0/3939428494 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa14c1cdd50 con 0x7fa14c0b87e0 2024-04-04T21:01:27.530 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.530+0000 7fa1437fe640 1 -- 172.21.15.106:0/3939428494 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 2) v1 ==== 44914+0+0 (secure 0 0 0) 0x7fa13401a070 con 0x7fa14c0b87e0 2024-04-04T21:01:27.547 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.546+0000 7fa1437fe640 1 -- 172.21.15.106:0/3939428494 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fa134010e00 con 0x7fa14c0b87e0 2024-04-04T21:01:27.547 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.546+0000 7fa1437fe640 10 client.4157.objecter ms_dispatch 0x7fa14c000b90 osd_map(10..10 src has 1..10) v4 2024-04-04T21:01:27.547 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.546+0000 7fa1437fe640 3 client.4157.objecter handle_osd_map got epochs [10,10] > 0 2024-04-04T21:01:27.547 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.546+0000 7fa1437fe640 3 client.4157.objecter handle_osd_map decoding full epoch 10 2024-04-04T21:01:27.547 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.546+0000 7fa1437fe640 20 client.4157.objecter dump_active .. 0 homeless 2024-04-04T21:01:27.686 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.686+0000 7fa1437fe640 1 -- 172.21.15.106:0/3939428494 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa134021020 con 0x7fa14c0b87e0 2024-04-04T21:01:27.686 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.686+0000 7fa1437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:27.686 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.686+0000 7fa1437fe640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:27.810 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.809+0000 7fa1515dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:27.810 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.809+0000 7fa1515dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:27.810 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.809+0000 7fa1515dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:27.810 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.809+0000 7fa1515dd640 1 -- 172.21.15.106:0/3939428494 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa14c0689e0 con 0x7fa14c0b87e0 2024-04-04T21:01:27.981 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.981+0000 7fa1437fe640 1 -- 172.21.15.106:0/3939428494 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mgrmap(e 3) v1 ==== 89616+0+0 (secure 0 0 0) 0x7fa13401b8d0 con 0x7fa14c0b87e0 2024-04-04T21:01:27.982 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.981+0000 7fa1437fe640 1 --2- 172.21.15.106:0/3939428494 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa12c06d380 0x7fa12c06f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:27.982 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.982+0000 7fa14affd640 1 --2- 172.21.15.106:0/3939428494 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa12c06d380 0x7fa12c06f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:27.982 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.982+0000 7fa14affd640 10 monclient: get_auth_request con 0x7fa12c06d380 auth_method 0 2024-04-04T21:01:27.982 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.982+0000 7fa1437fe640 10 client.4157.objecter ms_handle_connect 0x7fa12c06d380 2024-04-04T21:01:27.983 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:27.982+0000 7fa14affd640 1 --2- 172.21.15.106:0/3939428494 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa12c06d380 0x7fa12c06f770 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa13c036000 tx=0x7fa13c030000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:28.074 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:28.073+0000 7f6d5b8f4640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-04T21:01:28.163 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:01:28.162+0000 7f6d5b8f4640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-04T21:01:28.361 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.360+0000 7fa1437fe640 1 -- 172.21.15.106:0/3939428494 <== mon.2 v2:172.21.15.106:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fa13405b0b0 con 0x7fa14c0b87e0 2024-04-04T21:01:28.361 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.360+0000 7fa1437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:28.361 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.360+0000 7fa1437fe640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:01:28.362 INFO:teuthology.misc.health.smithi106.stdout: 2024-04-04T21:01:28.362 INFO:teuthology.misc.health.smithi106.stdout:{"epoch":10,"fsid":"8f89ce43-1bcb-41cf-9555-c40eeb258155","created":"2024-04-04T21:01:20.057083+0000","modified":"2024-04-04T21:01:25.642687+0000","last_up_change":"0.000000","last_in_change":"2024-04-04T21:01:25.642687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"28337134-5253-454a-bcda-47267b05b2c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ae336a0e-2a90-4323-9f0f-1701fffe6b66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"773087e0-78c9-4d14-bf0a-fb605c21f92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07da9513-c746-46d1-995f-6f8e91338b84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c47d034c-1db6-4332-bf24-722fb2846808","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"588caa62-e773-44ae-94e5-ece54fe980d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ef776e12-7c80-4095-b173-5b2d24996f95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"53aafc58-6463-4aeb-8395-e14fcfcee5de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-04T21:01:28.364 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.361+0000 7fa1515dd640 10 librados: watch_flush enter 2024-04-04T21:01:28.364 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 10 librados: watch_flush exit 2024-04-04T21:01:28.364 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 20 client.4157.objecter shutdown clearing up homeless session... 2024-04-04T21:01:28.364 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 10 client.4157.objecter successfully canceled tick 2024-04-04T21:01:28.364 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 1 -- 172.21.15.106:0/3939428494 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa12c06d380 msgr2=0x7fa12c06f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:28.364 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 1 --2- 172.21.15.106:0/3939428494 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa12c06d380 0x7fa12c06f770 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa13c036000 tx=0x7fa13c030000 comp rx=0 tx=0).stop 2024-04-04T21:01:28.364 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 10 monclient: shutdown 2024-04-04T21:01:28.364 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:28.364 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 1 -- 172.21.15.106:0/3939428494 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa14c0b87e0 msgr2=0x7fa14c1d82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:28.364 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 1 --2- 172.21.15.106:0/3939428494 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa14c0b87e0 0x7fa14c1d82f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa13400c410 tx=0x7fa13400c8d0 comp rx=0 tx=0).stop 2024-04-04T21:01:28.364 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 1 -- 172.21.15.106:0/3939428494 shutdown_connections 2024-04-04T21:01:28.365 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 1 --2- 172.21.15.106:0/3939428494 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa12c06d380 0x7fa12c06f770 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:28.365 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 1 --2- 172.21.15.106:0/3939428494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa14c13fc70 0x7fa14c1db5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:28.365 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 1 --2- 172.21.15.106:0/3939428494 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa14c0b87e0 0x7fa14c1d82f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:28.365 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 1 --2- 172.21.15.106:0/3939428494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa14c0aa300 0x7fa14c1465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:28.365 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 1 -- 172.21.15.106:0/3939428494 >> 172.21.15.106:0/3939428494 conn(0x7fa14c0727d0 msgr2=0x7fa14c13b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:28.365 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 1 -- 172.21.15.106:0/3939428494 shutdown_connections 2024-04-04T21:01:28.365 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 1 -- 172.21.15.106:0/3939428494 wait complete. 2024-04-04T21:01:28.365 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:28.362+0000 7fa1515dd640 1 librados: shutdown 2024-04-04T21:01:28.376 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-04T21:01:33.085 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:07.269 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.125 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:07.326 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.156 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:07.340 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.187 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:07.379 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.205 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:07.402 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.253 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:07.445 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.368 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:07.573 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.431 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:07.637 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.465 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:07.680 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.487 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:07.704 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.527 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:07.741 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.541 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:07.741 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.553 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:07.770 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.605 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:07.805 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.635 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:07.836 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.694 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-04T21:01:33.688+0000 5dc3480 -1 Falling back to public interface 2024-04-04T21:01:33.695 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:07.896 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:33.777 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:01:33.771+0000 5dc3480 -1 Falling back to public interface 2024-04-04T21:01:33.806 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-04-04T21:01:33.799+0000 5dc3480 -1 Falling back to public interface 2024-04-04T21:01:34.001 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:01:33.995+0000 5dc3480 -1 Falling back to public interface 2024-04-04T21:01:34.079 INFO:tasks.ceph.osd.3.smithi106.stderr:2024-04-04T21:01:34.070+0000 5dc3480 -1 Falling back to public interface 2024-04-04T21:01:34.113 INFO:tasks.ceph.osd.2.smithi106.stderr:2024-04-04T21:01:34.106+0000 5dc3480 -1 Falling back to public interface 2024-04-04T21:01:34.157 INFO:tasks.ceph.osd.0.smithi106.stderr:2024-04-04T21:01:34.150+0000 5dc3480 -1 Falling back to public interface 2024-04-04T21:01:34.160 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:08.345 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:34.270 INFO:tasks.ceph.osd.1.smithi106.stderr:2024-04-04T21:01:34.264+0000 5dc3480 -1 Falling back to public interface 2024-04-04T21:01:34.291 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:08.492 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:34.322 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:08.514 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:34.509 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:08.715 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:34.525 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:08.709 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:34.565 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:08.780 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:34.629 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:08.846 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:34.649 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:08.849 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:34.661 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:08.863 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:34.691 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:08.884 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:34.792 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:08.993 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:34.842 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:09.026 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:34.877 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:09.082 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:34.928 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:09.143 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:34.974 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:09.175 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.004 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:09.196 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.004 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:09.222 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.043 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:09.243 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.107 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:09.292 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.158 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:09.358 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.181 DEBUG:teuthology.orchestra.run.smithi106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-04T21:01:35.227 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:09.432 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.240 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:09.442 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.272 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.269+0000 7fd79f49a640 10 monclient: get_monmap_and_config 2024-04-04T21:01:35.272 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.269+0000 7fd79f49a640 10 monclient: build_initial_monmap 2024-04-04T21:01:35.272 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.269+0000 7fd79f49a640 10 monclient: monmap: 2024-04-04T21:01:35.273 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:01:35.273 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:35.273 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:01:35.270231+0000 2024-04-04T21:01:35.273 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:01:35.270231+0000 2024-04-04T21:01:35.273 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:35.273 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:35.273 INFO:teuthology.misc.health.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:35.273 INFO:teuthology.misc.health.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:35.273 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:35.273 INFO:teuthology.misc.health.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:35.273 INFO:teuthology.misc.health.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:35.273 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:35.273 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.269+0000 7fd79f49a640 1 Processor -- start 2024-04-04T21:01:35.273 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.269+0000 7fd79f49a640 1 -- start start 2024-04-04T21:01:35.274 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.269+0000 7fd79f49a640 10 monclient: init 2024-04-04T21:01:35.274 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.270+0000 7fd79f49a640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:35.274 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.270+0000 7fd79f49a640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-04-04T21:01:35.274 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.270+0000 7fd79f49a640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7980b87e0 0x7fd798142650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:35.274 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.270+0000 7fd79f49a640 10 monclient(hunting): picked mon.noname-b con 0x7fd7980b87e0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:35.274 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.270+0000 7fd79f49a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7980aa300 addr v1:172.21.15.203:6789/0 2024-04-04T21:01:35.274 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.270+0000 7fd79f49a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd79813fc70 addr v1:172.21.15.106:6789/0 2024-04-04T21:01:35.274 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.270+0000 7fd79f49a640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7980ab360 con 0x7fd79813fc70 2024-04-04T21:01:35.274 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.270+0000 7fd79f49a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7980ab640 con 0x7fd7980aa300 2024-04-04T21:01:35.274 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.270+0000 7fd79f49a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:35.274 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.270+0000 7fd79f49a640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7980b8bc0 con 0x7fd7980b87e0 2024-04-04T21:01:35.274 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.270+0000 7fd79f49a640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:35.274 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.270+0000 7fd79f49a640 10 monclient(hunting): authenticate will time out at 1344.537735s 2024-04-04T21:01:35.274 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.271+0000 7fd79d236640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7980b87e0 0x7fd798142650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:35.275 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.271+0000 7fd79da37640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fd79813fc70 0x7fd79814cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:40602/0 (socket says 172.21.15.106:40602) 2024-04-04T21:01:35.275 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.271+0000 7fd79da37640 1 -- 172.21.15.106:0/4268055163 learned_addr learned my addr 172.21.15.106:0/4268055163 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:01:35.275 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.271+0000 7fd79d236640 10 monclient(hunting): get_auth_request con 0x7fd7980b87e0 auth_method 0 2024-04-04T21:01:35.275 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.271+0000 7fd79d236640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:35.275 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.271+0000 7fd79d236640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:35.275 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.271+0000 7fd79d236640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:35.275 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.272+0000 7fd79d236640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:35.275 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.272+0000 7fd79d236640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:35.275 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.272+0000 7fd79d236640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:35.276 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:09.465 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.277 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd78ffff640 1 -- 172.21.15.106:0/4268055163 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816009765 0 0) 0x7fd7980ab640 con 0x7fd7980aa300 2024-04-04T21:01:35.277 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd78ffff640 10 monclient(hunting): _negotiate 2024-04-04T21:01:35.277 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd78ffff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:35.277 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd78ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:35.277 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd78ffff640 10 monclient(hunting): my global_id is 4213 2024-04-04T21:01:35.277 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd78ffff640 1 -- 172.21.15.106:0/4268055163 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd784003520 con 0x7fd7980aa300 2024-04-04T21:01:35.278 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd78ffff640 1 -- 172.21.15.106:0/4268055163 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49230233 0 0) 0x7fd7980ab360 con 0x7fd79813fc70 2024-04-04T21:01:35.278 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd78ffff640 10 monclient(hunting): _negotiate 2024-04-04T21:01:35.278 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd78ffff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:35.278 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd78ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:35.278 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd78ffff640 10 monclient(hunting): my global_id is 4188 2024-04-04T21:01:35.278 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd78ffff640 1 -- 172.21.15.106:0/4268055163 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7980ab640 con 0x7fd79813fc70 2024-04-04T21:01:35.278 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd79d236640 10 monclient(hunting): handle_auth_done global_id 4172 payload 274 2024-04-04T21:01:35.278 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd79d236640 1 -- 172.21.15.106:0/4268055163 >> v1:172.21.15.203:6789/0 conn(0x7fd7980aa300 legacy=0x7fd79813f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:35.278 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd79d236640 1 -- 172.21.15.106:0/4268055163 >> v1:172.21.15.106:6789/0 conn(0x7fd79813fc70 legacy=0x7fd79814cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:35.278 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd79d236640 10 monclient: _finish_hunting 0 2024-04-04T21:01:35.278 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd79d236640 1 monclient: found mon.noname-b 2024-04-04T21:01:35.278 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd79d236640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:35.278 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd79d236640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:01:35.278 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.273+0000 7fd79d236640 1 -- 172.21.15.106:0/4268055163 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd798140190 con 0x7fd7980b87e0 2024-04-04T21:01:35.279 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.275+0000 7fd79d236640 1 --2- 172.21.15.106:0/4268055163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7980b87e0 0x7fd798142650 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd790009dc0 tx=0x7fd79002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5dd81b85b68eed98 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:35.280 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd78ffff640 1 -- 172.21.15.106:0/4268055163 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790039070 con 0x7fd7980b87e0 2024-04-04T21:01:35.280 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd78ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:35.280 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd78ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:35.281 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd78ffff640 10 monclient: dump: 2024-04-04T21:01:35.281 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:35.281 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:35.281 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:35.281 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:35.281 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:35.281 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:35.281 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:35.281 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:35.281 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:35.281 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:35.281 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd78ffff640 10 monclient: _finish_auth 0 2024-04-04T21:01:35.281 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd78ffff640 10 monclient: _check_auth_tickets 2024-04-04T21:01:35.281 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd78ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:35.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd78ffff640 1 -- 172.21.15.106:0/4268055163 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79002cc40 con 0x7fd7980b87e0 2024-04-04T21:01:35.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd79f49a640 5 monclient: authenticate success, global_id 4172 2024-04-04T21:01:35.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd79f49a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:01:35.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd78ffff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:35.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd79f49a640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:35.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd79f49a640 4 set_mon_vals no callback set 2024-04-04T21:01:35.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd78ffff640 1 -- 172.21.15.106:0/4268055163 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790034440 con 0x7fd7980b87e0 2024-04-04T21:01:35.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd78ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:35.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd78ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:01:35.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.279+0000 7fd78ffff640 10 monclient: dump: 2024-04-04T21:01:35.282 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.280+0000 7fd79f49a640 10 monclient: shutdown 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.280+0000 7fd79f49a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.280+0000 7fd79f49a640 1 -- 172.21.15.106:0/4268055163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7980b87e0 msgr2=0x7fd798142650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.280+0000 7fd79f49a640 1 --2- 172.21.15.106:0/4268055163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7980b87e0 0x7fd798142650 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd790009dc0 tx=0x7fd79002be90 comp rx=0 tx=0).stop 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.280+0000 7fd79f49a640 1 -- 172.21.15.106:0/4268055163 shutdown_connections 2024-04-04T21:01:35.283 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.280+0000 7fd79f49a640 1 --2- 172.21.15.106:0/4268055163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7980b87e0 0x7fd798142650 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:35.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.280+0000 7fd79f49a640 1 -- 172.21.15.106:0/4268055163 >> 172.21.15.106:0/4268055163 conn(0x7fd7980727d0 msgr2=0x7fd798072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:35.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.280+0000 7fd79f49a640 1 -- 172.21.15.106:0/4268055163 shutdown_connections 2024-04-04T21:01:35.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.280+0000 7fd79f49a640 1 -- 172.21.15.106:0/4268055163 wait complete. 2024-04-04T21:01:35.288 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.280+0000 7fd79f49a640 10 monclient: build_initial_monmap 2024-04-04T21:01:35.288 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.280+0000 7fd79f49a640 10 monclient: monmap: 2024-04-04T21:01:35.288 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:01:35.289 INFO:teuthology.misc.health.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:35.289 INFO:teuthology.misc.health.smithi106.stderr:last_changed 0.000000 2024-04-04T21:01:35.289 INFO:teuthology.misc.health.smithi106.stderr:created 0.000000 2024-04-04T21:01:35.289 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:35.289 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:35.289 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:35.289 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:35.289 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:35.289 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:35.289 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.281+0000 7fd79f49a640 1 librados: starting msgr at 2024-04-04T21:01:35.289 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.281+0000 7fd79f49a640 1 librados: starting objecter 2024-04-04T21:01:35.289 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.281+0000 7fd79f49a640 1 Processor -- start 2024-04-04T21:01:35.289 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.281+0000 7fd79f49a640 1 -- start start 2024-04-04T21:01:35.289 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.281+0000 7fd79f49a640 1 librados: setting wanted keys 2024-04-04T21:01:35.290 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.281+0000 7fd79f49a640 1 librados: calling monclient init 2024-04-04T21:01:35.290 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.281+0000 7fd79f49a640 10 monclient: init 2024-04-04T21:01:35.290 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.281+0000 7fd79f49a640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:35.290 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.281+0000 7fd79f49a640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:01:35.290 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.281+0000 7fd79f49a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7980aa300 0x7fd798146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:35.290 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.281+0000 7fd79f49a640 10 monclient(hunting): picked mon.noname-a con 0x7fd7980aa300 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:35.290 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.281+0000 7fd79f49a640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7980b87e0 0x7fd798135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:35.290 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.281+0000 7fd79f49a640 10 monclient(hunting): picked mon.noname-c con 0x7fd7980b87e0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:35.290 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.281+0000 7fd79f49a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd79813fc70 0x7fd798135970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:35.290 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79f49a640 10 monclient(hunting): picked mon.noname-b con 0x7fd79813fc70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:35.290 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79f49a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:35.290 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79f49a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7980a9bc0 con 0x7fd7980aa300 2024-04-04T21:01:35.290 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79f49a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:35.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79f49a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7980a9a40 con 0x7fd79813fc70 2024-04-04T21:01:35.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79f49a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:35.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79f49a640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd798142b90 con 0x7fd7980b87e0 2024-04-04T21:01:35.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79f49a640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:35.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79f49a640 10 monclient(hunting): authenticate will time out at 1344.549227s 2024-04-04T21:01:35.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79ca35640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7980b87e0 0x7fd798135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:35.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79d236640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7980aa300 0x7fd798146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:35.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79da37640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd79813fc70 0x7fd798135970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:35.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79da37640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd79813fc70 0x7fd798135970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:52140/0 (socket says 172.21.15.106:52140) 2024-04-04T21:01:35.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79da37640 1 -- 172.21.15.106:0/111921925 learned_addr learned my addr 172.21.15.106:0/111921925 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:01:35.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79da37640 10 monclient(hunting): get_auth_request con 0x7fd79813fc70 auth_method 0 2024-04-04T21:01:35.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79da37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:35.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79da37640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:35.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79da37640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:35.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79ca35640 10 monclient(hunting): get_auth_request con 0x7fd7980b87e0 auth_method 0 2024-04-04T21:01:35.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79ca35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:35.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79ca35640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:35.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79ca35640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:35.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79d236640 10 monclient(hunting): get_auth_request con 0x7fd7980aa300 auth_method 0 2024-04-04T21:01:35.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79d236640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:35.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79d236640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:35.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.282+0000 7fd79d236640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:35.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.283+0000 7fd79da37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:35.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.283+0000 7fd79da37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:35.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.283+0000 7fd79da37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:35.293 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.283+0000 7fd79ca35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:35.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.283+0000 7fd79ca35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:35.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.283+0000 7fd79ca35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:35.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.283+0000 7fd79d236640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:35.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.283+0000 7fd79d236640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:35.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.283+0000 7fd79d236640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:35.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.285+0000 7fd79da37640 10 monclient(hunting): handle_auth_done global_id 4216 payload 824 2024-04-04T21:01:35.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.285+0000 7fd79da37640 1 -- 172.21.15.106:0/111921925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7980b87e0 msgr2=0x7fd798135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:35.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.285+0000 7fd79da37640 1 --2- 172.21.15.106:0/111921925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7980b87e0 0x7fd798135430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:35.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.286+0000 7fd79da37640 1 -- 172.21.15.106:0/111921925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7980aa300 msgr2=0x7fd798146400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:35.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.286+0000 7fd79da37640 1 --2- 172.21.15.106:0/111921925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7980aa300 0x7fd798146400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:35.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.286+0000 7fd79da37640 10 monclient: _finish_hunting 0 2024-04-04T21:01:35.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.286+0000 7fd79da37640 1 monclient: found mon.noname-b 2024-04-04T21:01:35.295 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.286+0000 7fd79da37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:35.295 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.286+0000 7fd79da37640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:01:35.295 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.286+0000 7fd79da37640 1 -- 172.21.15.106:0/111921925 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd798145720 con 0x7fd79813fc70 2024-04-04T21:01:35.295 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.288+0000 7fd79da37640 1 --2- 172.21.15.106:0/111921925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd79813fc70 0x7fd798135970 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c3c0 tx=0x7fd79400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:35.295 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.288+0000 7fd78dffb640 10 client.?.objecter ms_handle_connect 0x7fd79813fc70 2024-04-04T21:01:35.295 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.288+0000 7fd78dffb640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:35.296 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd78dffb640 1 -- 172.21.15.106:0/111921925 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794017070 con 0x7fd79813fc70 2024-04-04T21:01:35.296 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd78dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:35.296 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd78dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:35.297 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd78dffb640 10 monclient: dump: 2024-04-04T21:01:35.297 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:35.297 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:35.297 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:35.297 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:35.297 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:35.297 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:35.297 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:35.298 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:35.298 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:35.298 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:35.298 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd78dffb640 10 monclient: _finish_auth 0 2024-04-04T21:01:35.298 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd78dffb640 10 monclient: _check_auth_tickets 2024-04-04T21:01:35.298 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd78dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:35.299 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd78dffb640 1 -- 172.21.15.106:0/111921925 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd794005ce0 con 0x7fd79813fc70 2024-04-04T21:01:35.299 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd78dffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:35.299 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd78dffb640 1 -- 172.21.15.106:0/111921925 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7940072f0 con 0x7fd79813fc70 2024-04-04T21:01:35.299 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd78dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:35.299 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd78dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:01:35.299 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd78dffb640 10 monclient: dump: 2024-04-04T21:01:35.299 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:35.299 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:35.299 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:35.299 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:35.300 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:35.300 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:35.300 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:35.300 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:35.301 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:35.301 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:35.301 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd78f7fe640 4 set_mon_vals no callback set 2024-04-04T21:01:35.301 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd79f49a640 5 monclient: authenticate success, global_id 4216 2024-04-04T21:01:35.301 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd79f49a640 10 monclient: _renew_subs 2024-04-04T21:01:35.301 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd79f49a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:35.301 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd79f49a640 1 -- 172.21.15.106:0/111921925 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7980b2e40 con 0x7fd79813fc70 2024-04-04T21:01:35.301 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd79f49a640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:35.302 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd79f49a640 10 monclient: _renew_subs 2024-04-04T21:01:35.302 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd79f49a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:35.302 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd79f49a640 1 -- 172.21.15.106:0/111921925 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd798145c70 con 0x7fd79813fc70 2024-04-04T21:01:35.302 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.291+0000 7fd79f49a640 1 librados: init done 2024-04-04T21:01:35.303 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.292+0000 7fd79f49a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:35.303 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.292+0000 7fd79f49a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:35.303 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.292+0000 7fd79f49a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:35.303 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.292+0000 7fd79f49a640 1 -- 172.21.15.106:0/111921925 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd79806dd50 con 0x7fd79813fc70 2024-04-04T21:01:35.310 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.310+0000 7fd78dffb640 1 -- 172.21.15.106:0/111921925 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fd79401a070 con 0x7fd79813fc70 2024-04-04T21:01:35.311 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.310+0000 7fd78dffb640 1 --2- 172.21.15.106:0/111921925 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd77c06ce50 0x7fd77c06f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:35.311 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.310+0000 7fd79d236640 1 --2- 172.21.15.106:0/111921925 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd77c06ce50 0x7fd77c06f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:35.311 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.310+0000 7fd79d236640 10 monclient: get_auth_request con 0x7fd77c06ce50 auth_method 0 2024-04-04T21:01:35.311 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.311+0000 7fd79d236640 1 --2- 172.21.15.106:0/111921925 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd77c06ce50 0x7fd77c06f310 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd790036000 tx=0x7fd790030000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:35.311 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.311+0000 7fd78dffb640 10 client.4216.objecter ms_handle_connect 0x7fd77c06ce50 2024-04-04T21:01:35.313 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.312+0000 7fd78dffb640 1 -- 172.21.15.106:0/111921925 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fd794056d30 con 0x7fd79813fc70 2024-04-04T21:01:35.313 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.312+0000 7fd78dffb640 10 client.4216.objecter ms_dispatch 0x7fd798000b90 osd_map(10..10 src has 1..10) v4 2024-04-04T21:01:35.313 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.312+0000 7fd78dffb640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2024-04-04T21:01:35.313 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.312+0000 7fd78dffb640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2024-04-04T21:01:35.313 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.312+0000 7fd78dffb640 20 client.4216.objecter dump_active .. 0 homeless 2024-04-04T21:01:35.324 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:09.539 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.361 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:09.579 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.366 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:09.566 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.379 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:09.564 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.513 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:09.715 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.518 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:09.719 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.519 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:09.725 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.550 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:09.743 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.570 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.570+0000 7fd78dffb640 1 -- 172.21.15.106:0/111921925 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd794021020 con 0x7fd79813fc70 2024-04-04T21:01:35.571 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.570+0000 7fd78dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:35.571 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.570+0000 7fd78dffb640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:35.613 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:09.827 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.644 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:09.828 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.646 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:09.847 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.672 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:09.889 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.769 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.768+0000 7fd79f49a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:35.769 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.768+0000 7fd79f49a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:35.769 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.768+0000 7fd79f49a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:01:35.769 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.768+0000 7fd79f49a640 1 -- 172.21.15.106:0/111921925 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd798067de0 con 0x7fd79813fc70 2024-04-04T21:01:35.778 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:09.980 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.805 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:10.006 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.814 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.813+0000 7fd78dffb640 1 -- 172.21.15.106:0/111921925 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fd79405bf00 con 0x7fd79813fc70 2024-04-04T21:01:35.814 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.813+0000 7fd78dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:35.814 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.813+0000 7fd78dffb640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:01:35.814 INFO:teuthology.misc.health.smithi106.stdout: 2024-04-04T21:01:35.814 INFO:teuthology.misc.health.smithi106.stdout:{"epoch":10,"fsid":"8f89ce43-1bcb-41cf-9555-c40eeb258155","created":"2024-04-04T21:01:20.057083+0000","modified":"2024-04-04T21:01:25.642687+0000","last_up_change":"0.000000","last_in_change":"2024-04-04T21:01:25.642687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"28337134-5253-454a-bcda-47267b05b2c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ae336a0e-2a90-4323-9f0f-1701fffe6b66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"773087e0-78c9-4d14-bf0a-fb605c21f92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07da9513-c746-46d1-995f-6f8e91338b84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c47d034c-1db6-4332-bf24-722fb2846808","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"588caa62-e773-44ae-94e5-ece54fe980d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ef776e12-7c80-4095-b173-5b2d24996f95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"53aafc58-6463-4aeb-8395-e14fcfcee5de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-04T21:01:35.817 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 10 librados: watch_flush enter 2024-04-04T21:01:35.817 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 10 librados: watch_flush exit 2024-04-04T21:01:35.817 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 20 client.4216.objecter shutdown clearing up homeless session... 2024-04-04T21:01:35.817 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 10 client.4216.objecter successfully canceled tick 2024-04-04T21:01:35.817 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 1 -- 172.21.15.106:0/111921925 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd77c06ce50 msgr2=0x7fd77c06f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:35.818 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 1 --2- 172.21.15.106:0/111921925 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd77c06ce50 0x7fd77c06f310 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd790036000 tx=0x7fd790030000 comp rx=0 tx=0).stop 2024-04-04T21:01:35.818 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 10 monclient: shutdown 2024-04-04T21:01:35.818 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:35.818 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 1 -- 172.21.15.106:0/111921925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd79813fc70 msgr2=0x7fd798135970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:35.818 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 1 --2- 172.21.15.106:0/111921925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd79813fc70 0x7fd798135970 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c3c0 tx=0x7fd79400c880 comp rx=0 tx=0).stop 2024-04-04T21:01:35.818 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 1 -- 172.21.15.106:0/111921925 shutdown_connections 2024-04-04T21:01:35.818 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 1 --2- 172.21.15.106:0/111921925 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd77c06ce50 0x7fd77c06f310 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:35.818 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 1 --2- 172.21.15.106:0/111921925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd79813fc70 0x7fd798135970 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:35.818 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 1 --2- 172.21.15.106:0/111921925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7980b87e0 0x7fd798135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:35.818 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 1 --2- 172.21.15.106:0/111921925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7980aa300 0x7fd798146400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:35.818 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 1 -- 172.21.15.106:0/111921925 >> 172.21.15.106:0/111921925 conn(0x7fd7980727d0 msgr2=0x7fd79813b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:35.818 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 1 -- 172.21.15.106:0/111921925 shutdown_connections 2024-04-04T21:01:35.818 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 1 -- 172.21.15.106:0/111921925 wait complete. 2024-04-04T21:01:35.818 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:35.816+0000 7fd79f49a640 1 librados: shutdown 2024-04-04T21:01:35.821 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:10.014 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.826 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-04T21:01:35.829 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:10.034 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.905 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:10.120 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.912 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:10.113 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.917 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:10.102 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:35.979 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:10.197 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.046 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:10.248 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.087 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:10.280 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.103 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:10.304 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.118 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:10.324 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.185 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:10.370 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.189 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:10.389 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.205 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:10.420 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.270 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:10.488 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.314 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:10.516 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.352 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:10.544 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.395 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:10.596 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.405 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:10.610 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.454 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:10.639 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.456 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:10.657 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.492 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:10.707 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.557 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:10.775 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.582 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:10.784 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.623 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:10.816 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.683 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:10.884 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.692 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:10.897 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.724 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:10.924 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.779 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:10.994 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.844 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:11.062 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.948 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:11.132 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.970 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:11.171 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.985 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:11.191 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:36.990 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:11.191 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.002 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:11.186 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.067 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:11.282 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.081 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:11.283 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.096 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:11.289 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.137 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:11.338 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.138 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:11.356 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.152 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:11.344 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.262 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:11.463 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.463 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:11.663 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.517 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:11.717 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.563 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:11.769 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.592 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:11.807 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.639 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:11.845 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.654 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:11.869 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.671 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:11.888 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.725 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:11.943 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.817 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:12.018 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:37.872 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:12.072 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:42.455 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:16.657 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:42.634 DEBUG:teuthology.orchestra.run.smithi106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-04T21:01:42.637 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:16.821 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:42.645 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:16.837 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:42.723 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.723+0000 7fdf09bcc640 10 monclient: get_monmap_and_config 2024-04-04T21:01:42.723 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.723+0000 7fdf09bcc640 10 monclient: build_initial_monmap 2024-04-04T21:01:42.723 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.723+0000 7fdf09bcc640 10 monclient: monmap: 2024-04-04T21:01:42.724 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:01:42.724 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:42.724 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:01:42.724187+0000 2024-04-04T21:01:42.724 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:01:42.724187+0000 2024-04-04T21:01:42.724 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:42.724 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:42.724 INFO:teuthology.misc.health.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:42.724 INFO:teuthology.misc.health.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:42.724 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:42.724 INFO:teuthology.misc.health.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:42.724 INFO:teuthology.misc.health.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:42.724 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:42.727 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.723+0000 7fdf09bcc640 1 Processor -- start 2024-04-04T21:01:42.727 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.723+0000 7fdf09bcc640 1 -- start start 2024-04-04T21:01:42.727 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.723+0000 7fdf09bcc640 10 monclient: init 2024-04-04T21:01:42.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:42.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-04-04T21:01:42.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdf040b51a0 0x7fdf040a9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:42.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 10 monclient(hunting): picked mon.noname-a con 0x7fdf040b51a0 addr v2:172.21.15.106:3300/0 2024-04-04T21:01:42.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdf040b1d60 0x7fdf041425a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:42.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 10 monclient(hunting): picked mon.noname-c con 0x7fdf040b1d60 addr v2:172.21.15.203:3300/0 2024-04-04T21:01:42.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf040aa2e0 0x7fdf0413f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:42.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 10 monclient(hunting): picked mon.noname-b con 0x7fdf040aa2e0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:42.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:42.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf040b5580 con 0x7fdf040b51a0 2024-04-04T21:01:42.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:42.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf040a9870 con 0x7fdf040b1d60 2024-04-04T21:01:42.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:42.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040ab3d0 con 0x7fdf040aa2e0 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.724+0000 7fdf09bcc640 10 monclient(hunting): authenticate will time out at 1351.991751s 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf037fe640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdf040b51a0 0x7fdf040a9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf03fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf040aa2e0 0x7fdf0413f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf037fe640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdf040b51a0 0x7fdf040a9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35276/0 (socket says 172.21.15.106:35276) 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf02ffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdf040b1d60 0x7fdf041425a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf037fe640 1 -- 172.21.15.106:0/3857504613 learned_addr learned my addr 172.21.15.106:0/3857504613 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf037fe640 10 monclient(hunting): get_auth_request con 0x7fdf040b51a0 auth_method 0 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf037fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf03fff640 10 monclient(hunting): get_auth_request con 0x7fdf040aa2e0 auth_method 0 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf03fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:42.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:42.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf02ffd640 10 monclient(hunting): get_auth_request con 0x7fdf040b1d60 auth_method 0 2024-04-04T21:01:42.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:42.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf02ffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:42.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.725+0000 7fdf02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:42.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.726+0000 7fdf03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:42.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.726+0000 7fdf03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:42.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.726+0000 7fdf03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:42.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.726+0000 7fdf037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:42.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.726+0000 7fdf037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:42.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.726+0000 7fdf037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:42.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.726+0000 7fdf02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:42.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.726+0000 7fdf02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:42.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.726+0000 7fdf02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:42.732 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.728+0000 7fdf03fff640 10 monclient(hunting): handle_auth_done global_id 4178 payload 274 2024-04-04T21:01:42.732 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.728+0000 7fdf03fff640 1 -- 172.21.15.106:0/3857504613 >> v2:172.21.15.203:3300/0 conn(0x7fdf040b1d60 msgr2=0x7fdf041425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:01:42.732 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.728+0000 7fdf03fff640 1 --2- 172.21.15.106:0/3857504613 >> v2:172.21.15.203:3300/0 conn(0x7fdf040b1d60 0x7fdf041425a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:42.732 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.728+0000 7fdf03fff640 1 -- 172.21.15.106:0/3857504613 >> v2:172.21.15.106:3300/0 conn(0x7fdf040b51a0 msgr2=0x7fdf040a9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:01:42.732 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.728+0000 7fdf03fff640 1 --2- 172.21.15.106:0/3857504613 >> v2:172.21.15.106:3300/0 conn(0x7fdf040b51a0 0x7fdf040a9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:42.732 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.728+0000 7fdf03fff640 10 monclient: _finish_hunting 0 2024-04-04T21:01:42.732 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.728+0000 7fdf03fff640 1 monclient: found mon.noname-b 2024-04-04T21:01:42.732 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.728+0000 7fdf03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:42.732 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.728+0000 7fdf03fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:01:42.732 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.728+0000 7fdf03fff640 1 -- 172.21.15.106:0/3857504613 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf04142ae0 con 0x7fdf040aa2e0 2024-04-04T21:01:42.732 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.730+0000 7fdf03fff640 1 --2- 172.21.15.106:0/3857504613 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf040aa2e0 0x7fdf0413f750 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdf040aa9e0 tx=0x7fdefc02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7ae665356c959f server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:42.740 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf027fc640 1 -- 172.21.15.106:0/3857504613 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc038070 con 0x7fdf040aa2e0 2024-04-04T21:01:42.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:42.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:42.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf027fc640 10 monclient: dump: 2024-04-04T21:01:42.741 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:42.741 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:42.741 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:42.741 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:42.741 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:42.741 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:42.741 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:42.741 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:42.741 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:42.741 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:42.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf027fc640 10 monclient: _finish_auth 0 2024-04-04T21:01:42.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf027fc640 10 monclient: _check_auth_tickets 2024-04-04T21:01:42.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:42.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf027fc640 1 -- 172.21.15.106:0/3857504613 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdefc031700 con 0x7fdf040aa2e0 2024-04-04T21:01:42.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf027fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:42.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf027fc640 1 -- 172.21.15.106:0/3857504613 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc031ad0 con 0x7fdf040aa2e0 2024-04-04T21:01:42.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:42.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:01:42.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf027fc640 10 monclient: dump: 2024-04-04T21:01:42.742 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:42.742 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:42.742 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:42.742 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:42.742 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:42.743 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:42.743 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:42.743 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:42.743 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:42.743 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:42.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf09bcc640 5 monclient: authenticate success, global_id 4178 2024-04-04T21:01:42.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf09bcc640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:42.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf09bcc640 4 set_mon_vals no callback set 2024-04-04T21:01:42.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf09bcc640 10 monclient: shutdown 2024-04-04T21:01:42.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf09bcc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:42.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf09bcc640 1 -- 172.21.15.106:0/3857504613 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf040aa2e0 msgr2=0x7fdf0413f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:42.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf09bcc640 1 --2- 172.21.15.106:0/3857504613 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf040aa2e0 0x7fdf0413f750 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdf040aa9e0 tx=0x7fdefc02b7a0 comp rx=0 tx=0).stop 2024-04-04T21:01:42.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf09bcc640 1 -- 172.21.15.106:0/3857504613 shutdown_connections 2024-04-04T21:01:42.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf09bcc640 1 --2- 172.21.15.106:0/3857504613 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf040aa2e0 0x7fdf0413f750 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:42.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf09bcc640 1 --2- 172.21.15.106:0/3857504613 >> v2:172.21.15.203:3300/0 conn(0x7fdf040b1d60 0x7fdf041425a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:42.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.734+0000 7fdf09bcc640 1 --2- 172.21.15.106:0/3857504613 >> v2:172.21.15.106:3300/0 conn(0x7fdf040b51a0 0x7fdf040a9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:42.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.735+0000 7fdf09bcc640 1 -- 172.21.15.106:0/3857504613 >> 172.21.15.106:0/3857504613 conn(0x7fdf040727a0 msgr2=0x7fdf04072bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:42.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.735+0000 7fdf09bcc640 1 -- 172.21.15.106:0/3857504613 shutdown_connections 2024-04-04T21:01:42.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.735+0000 7fdf09bcc640 1 -- 172.21.15.106:0/3857504613 wait complete. 2024-04-04T21:01:42.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.736+0000 7fdf09bcc640 10 monclient: build_initial_monmap 2024-04-04T21:01:42.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.736+0000 7fdf09bcc640 10 monclient: monmap: 2024-04-04T21:01:42.744 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:01:42.744 INFO:teuthology.misc.health.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:42.744 INFO:teuthology.misc.health.smithi106.stderr:last_changed 0.000000 2024-04-04T21:01:42.744 INFO:teuthology.misc.health.smithi106.stderr:created 0.000000 2024-04-04T21:01:42.744 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:42.744 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:42.744 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.736+0000 7fdf09bcc640 1 librados: starting msgr at 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.736+0000 7fdf09bcc640 1 librados: starting objecter 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.736+0000 7fdf09bcc640 1 Processor -- start 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.736+0000 7fdf09bcc640 1 -- start start 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.736+0000 7fdf09bcc640 1 librados: setting wanted keys 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.736+0000 7fdf09bcc640 1 librados: calling monclient init 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.736+0000 7fdf09bcc640 10 monclient: init 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf040aa2e0 0x7fdf040b8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 10 monclient(hunting): picked mon.noname-b con 0x7fdf040aa2e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:42.745 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf040b1d60 0x7fdf041459e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:42.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 10 monclient(hunting): picked mon.noname-a con 0x7fdf040b1d60 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:42.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf040b51a0 0x7fdf041db120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:42.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 10 monclient(hunting): picked mon.noname-c con 0x7fdf040b51a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:42.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:42.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf04140060 con 0x7fdf040b1d60 2024-04-04T21:01:42.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:42.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0413fee0 con 0x7fdf040aa2e0 2024-04-04T21:01:42.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:42.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf041401e0 con 0x7fdf040b51a0 2024-04-04T21:01:42.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:42.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf09bcc640 10 monclient(hunting): authenticate will time out at 1352.004644s 2024-04-04T21:01:42.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.737+0000 7fdf037fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf040aa2e0 0x7fdf040b8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:42.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf02ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf040b1d60 0x7fdf041459e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:42.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf037fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf040aa2e0 0x7fdf040b8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:52154/0 (socket says 172.21.15.106:52154) 2024-04-04T21:01:42.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf037fe640 1 -- 172.21.15.106:0/4144320000 learned_addr learned my addr 172.21.15.106:0/4144320000 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:01:42.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf037fe640 10 monclient(hunting): get_auth_request con 0x7fdf040aa2e0 auth_method 0 2024-04-04T21:01:42.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:42.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf037fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:42.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:42.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf03fff640 1 --2- 172.21.15.106:0/4144320000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf040b51a0 0x7fdf041db120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:42.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf03fff640 10 monclient(hunting): get_auth_request con 0x7fdf040b51a0 auth_method 0 2024-04-04T21:01:42.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:42.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf03fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:42.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:42.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf02ffd640 10 monclient(hunting): get_auth_request con 0x7fdf040b1d60 auth_method 0 2024-04-04T21:01:42.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:42.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf02ffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:42.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.738+0000 7fdf02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:42.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.739+0000 7fdf037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:42.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.739+0000 7fdf037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:42.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.739+0000 7fdf037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:42.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.739+0000 7fdf03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:42.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.739+0000 7fdf03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:42.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.739+0000 7fdf03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:42.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.739+0000 7fdf02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:42.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.739+0000 7fdf02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:42.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.739+0000 7fdf02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:42.752 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.741+0000 7fdf03fff640 10 monclient(hunting): handle_auth_done global_id 4181 payload 824 2024-04-04T21:01:42.752 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.741+0000 7fdf03fff640 1 -- 172.21.15.106:0/4144320000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf040aa2e0 msgr2=0x7fdf040b8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:42.752 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.741+0000 7fdf03fff640 1 --2- 172.21.15.106:0/4144320000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf040aa2e0 0x7fdf040b8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:42.753 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.741+0000 7fdf03fff640 1 -- 172.21.15.106:0/4144320000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf040b1d60 msgr2=0x7fdf041459e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:42.753 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.741+0000 7fdf03fff640 1 --2- 172.21.15.106:0/4144320000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf040b1d60 0x7fdf041459e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:42.753 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.741+0000 7fdf03fff640 10 monclient: _finish_hunting 0 2024-04-04T21:01:42.753 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.741+0000 7fdf03fff640 1 monclient: found mon.noname-c 2024-04-04T21:01:42.753 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.741+0000 7fdf03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:42.753 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.741+0000 7fdf03fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:01:42.753 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.741+0000 7fdf03fff640 1 -- 172.21.15.106:0/4144320000 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf04146410 con 0x7fdf040b51a0 2024-04-04T21:01:42.753 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.744+0000 7fdf03fff640 1 --2- 172.21.15.106:0/4144320000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf040b51a0 0x7fdf041db120 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdefc033000 tx=0x7fdefc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:42.753 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.744+0000 7fdefbfff640 10 client.?.objecter ms_handle_connect 0x7fdf040b51a0 2024-04-04T21:01:42.753 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.744+0000 7fdefbfff640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:42.753 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdefbfff640 1 -- 172.21.15.106:0/4144320000 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc042070 con 0x7fdf040b51a0 2024-04-04T21:01:42.753 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdefbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:42.753 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdefbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:01:42.753 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdefbfff640 10 monclient: dump: 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdefbfff640 10 monclient: _finish_auth 0 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdefbfff640 10 monclient: _check_auth_tickets 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdefbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdefbfff640 1 -- 172.21.15.106:0/4144320000 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdefc03d890 con 0x7fdf040b51a0 2024-04-04T21:01:42.754 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdf09bcc640 5 monclient: authenticate success, global_id 4181 2024-04-04T21:01:42.755 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdf09bcc640 10 monclient: _renew_subs 2024-04-04T21:01:42.755 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdf09bcc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:42.755 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdf09bcc640 1 -- 172.21.15.106:0/4144320000 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf04145f20 con 0x7fdf040b51a0 2024-04-04T21:01:42.755 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdefbfff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:42.755 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdefbfff640 1 -- 172.21.15.106:0/4144320000 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc038040 con 0x7fdf040b51a0 2024-04-04T21:01:42.755 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdefbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:42.755 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdefbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:01:42.755 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdefbfff640 10 monclient: dump: 2024-04-04T21:01:42.755 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:42.755 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:42.755 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:42.755 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:42.755 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:42.755 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:42.756 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:42.756 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:42.756 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:42.756 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:42.756 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdf09bcc640 10 client.4181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:42.756 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdf09bcc640 10 monclient: _renew_subs 2024-04-04T21:01:42.756 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdf09bcc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:42.756 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdf09bcc640 1 -- 172.21.15.106:0/4144320000 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf041db840 con 0x7fdf040b51a0 2024-04-04T21:01:42.756 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdf01ffb640 4 set_mon_vals no callback set 2024-04-04T21:01:42.756 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.747+0000 7fdf09bcc640 1 librados: init done 2024-04-04T21:01:42.756 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.749+0000 7fdf09bcc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:42.756 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.749+0000 7fdf09bcc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:42.756 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.749+0000 7fdf09bcc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:42.756 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:42.749+0000 7fdf09bcc640 1 -- 172.21.15.106:0/4144320000 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0406a560 con 0x7fdf040b51a0 2024-04-04T21:01:43.071 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.070+0000 7fdefbfff640 1 -- 172.21.15.106:0/4144320000 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fdefc03b070 con 0x7fdf040b51a0 2024-04-04T21:01:43.071 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.071+0000 7fdefbfff640 1 --2- 172.21.15.106:0/4144320000 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdee406cdd0 0x7fdee406f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:43.071 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.071+0000 7fdf037fe640 1 --2- 172.21.15.106:0/4144320000 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdee406cdd0 0x7fdee406f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:43.071 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.071+0000 7fdf037fe640 10 monclient: get_auth_request con 0x7fdee406cdd0 auth_method 0 2024-04-04T21:01:43.072 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.071+0000 7fdefbfff640 10 client.4181.objecter ms_handle_connect 0x7fdee406cdd0 2024-04-04T21:01:43.072 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.071+0000 7fdf037fe640 1 --2- 172.21.15.106:0/4144320000 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdee406cdd0 0x7fdee406f290 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdef4011420 tx=0x7fdef400b040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:43.092 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.091+0000 7fdefbfff640 1 -- 172.21.15.106:0/4144320000 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fdefc0aa0e0 con 0x7fdf040b51a0 2024-04-04T21:01:43.092 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.091+0000 7fdefbfff640 10 client.4181.objecter ms_dispatch 0x7fdf04000b90 osd_map(10..10 src has 1..10) v4 2024-04-04T21:01:43.092 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.091+0000 7fdefbfff640 3 client.4181.objecter handle_osd_map got epochs [10,10] > 0 2024-04-04T21:01:43.092 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.091+0000 7fdefbfff640 3 client.4181.objecter handle_osd_map decoding full epoch 10 2024-04-04T21:01:43.092 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.091+0000 7fdefbfff640 20 client.4181.objecter dump_active .. 0 homeless 2024-04-04T21:01:43.092 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.091+0000 7fdefbfff640 1 -- 172.21.15.106:0/4144320000 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdefc030070 con 0x7fdf040b51a0 2024-04-04T21:01:43.092 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.091+0000 7fdefbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:43.092 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.091+0000 7fdefbfff640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:43.111 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:17.312 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:43.261 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:17.467 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:43.326 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.325+0000 7fdf09bcc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:43.326 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.325+0000 7fdf09bcc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:43.326 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.325+0000 7fdf09bcc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:43.326 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.325+0000 7fdf09bcc640 1 -- 172.21.15.106:0/4144320000 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf0413f0b0 con 0x7fdf040b51a0 2024-04-04T21:01:43.343 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.343+0000 7fdefbfff640 1 -- 172.21.15.106:0/4144320000 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fdefc07a4d0 con 0x7fdf040b51a0 2024-04-04T21:01:43.344 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.343+0000 7fdefbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:43.344 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.343+0000 7fdefbfff640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:01:43.344 INFO:teuthology.misc.health.smithi106.stdout: 2024-04-04T21:01:43.344 INFO:teuthology.misc.health.smithi106.stdout:{"epoch":10,"fsid":"8f89ce43-1bcb-41cf-9555-c40eeb258155","created":"2024-04-04T21:01:20.057083+0000","modified":"2024-04-04T21:01:25.642687+0000","last_up_change":"0.000000","last_in_change":"2024-04-04T21:01:25.642687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"28337134-5253-454a-bcda-47267b05b2c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ae336a0e-2a90-4323-9f0f-1701fffe6b66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"773087e0-78c9-4d14-bf0a-fb605c21f92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07da9513-c746-46d1-995f-6f8e91338b84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c47d034c-1db6-4332-bf24-722fb2846808","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"588caa62-e773-44ae-94e5-ece54fe980d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ef776e12-7c80-4095-b173-5b2d24996f95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"53aafc58-6463-4aeb-8395-e14fcfcee5de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-04T21:01:43.348 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.345+0000 7fdf09bcc640 10 librados: watch_flush enter 2024-04-04T21:01:43.348 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.345+0000 7fdf09bcc640 10 librados: watch_flush exit 2024-04-04T21:01:43.348 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.345+0000 7fdf09bcc640 20 client.4181.objecter shutdown clearing up homeless session... 2024-04-04T21:01:43.348 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.345+0000 7fdf09bcc640 10 client.4181.objecter successfully canceled tick 2024-04-04T21:01:43.348 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.345+0000 7fdf09bcc640 1 -- 172.21.15.106:0/4144320000 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdee406cdd0 msgr2=0x7fdee406f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:43.348 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.345+0000 7fdf09bcc640 1 --2- 172.21.15.106:0/4144320000 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdee406cdd0 0x7fdee406f290 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdef4011420 tx=0x7fdef400b040 comp rx=0 tx=0).stop 2024-04-04T21:01:43.348 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.345+0000 7fdf09bcc640 10 monclient: shutdown 2024-04-04T21:01:43.348 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.345+0000 7fdf09bcc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:43.349 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.345+0000 7fdf09bcc640 1 -- 172.21.15.106:0/4144320000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf040b51a0 msgr2=0x7fdf041db120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:43.349 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.345+0000 7fdf09bcc640 1 --2- 172.21.15.106:0/4144320000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf040b51a0 0x7fdf041db120 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdefc033000 tx=0x7fdefc0046c0 comp rx=0 tx=0).stop 2024-04-04T21:01:43.349 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.346+0000 7fdf09bcc640 1 -- 172.21.15.106:0/4144320000 shutdown_connections 2024-04-04T21:01:43.349 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.346+0000 7fdf09bcc640 1 --2- 172.21.15.106:0/4144320000 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdee406cdd0 0x7fdee406f290 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:43.349 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.346+0000 7fdf09bcc640 1 --2- 172.21.15.106:0/4144320000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf040b51a0 0x7fdf041db120 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:43.349 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.346+0000 7fdf09bcc640 1 --2- 172.21.15.106:0/4144320000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf040b1d60 0x7fdf041459e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:43.349 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.346+0000 7fdf09bcc640 1 --2- 172.21.15.106:0/4144320000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf040aa2e0 0x7fdf040b8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:43.349 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.346+0000 7fdf09bcc640 1 -- 172.21.15.106:0/4144320000 >> 172.21.15.106:0/4144320000 conn(0x7fdf040727a0 msgr2=0x7fdf04074580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:43.349 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.347+0000 7fdf09bcc640 1 -- 172.21.15.106:0/4144320000 shutdown_connections 2024-04-04T21:01:43.349 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.347+0000 7fdf09bcc640 1 -- 172.21.15.106:0/4144320000 wait complete. 2024-04-04T21:01:43.350 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:43.347+0000 7fdf09bcc640 1 librados: shutdown 2024-04-04T21:01:43.362 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-04T21:01:43.488 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:17.706 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:43.790 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:17.989 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:43.841 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:18.055 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:47.225 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:21.410 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:47.229 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:21.414 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:47.656 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:21.857 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:47.662 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:21.864 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:47.836 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:22.028 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:47.841 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:22.034 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:48.407 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:22.607 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:48.411 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:22.611 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:48.464 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:22.669 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:48.470 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:22.676 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:48.661 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:22.878 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:48.665 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:22.882 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:49.094 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:23.309 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:49.098 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:23.313 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:49.184 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:23.384 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:49.190 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:23.391 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:50.170 DEBUG:teuthology.orchestra.run.smithi106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-04T21:01:50.259 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.258+0000 7fd67fd6a640 10 monclient: get_monmap_and_config 2024-04-04T21:01:50.259 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.258+0000 7fd67fd6a640 10 monclient: build_initial_monmap 2024-04-04T21:01:50.259 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.258+0000 7fd67fd6a640 10 monclient: monmap: 2024-04-04T21:01:50.259 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:01:50.259 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:50.259 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:01:50.259641+0000 2024-04-04T21:01:50.259 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:01:50.259641+0000 2024-04-04T21:01:50.259 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:50.260 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:50.260 INFO:teuthology.misc.health.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:50.260 INFO:teuthology.misc.health.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:50.260 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:50.260 INFO:teuthology.misc.health.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:50.260 INFO:teuthology.misc.health.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:50.260 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:50.260 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.259+0000 7fd67fd6a640 1 Processor -- start 2024-04-04T21:01:50.261 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.259+0000 7fd67fd6a640 1 -- start start 2024-04-04T21:01:50.261 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.259+0000 7fd67fd6a640 10 monclient: init 2024-04-04T21:01:50.262 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.259+0000 7fd67fd6a640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:50.262 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.259+0000 7fd67fd6a640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-04-04T21:01:50.262 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.259+0000 7fd67fd6a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6780b51a0 addr v1:172.21.15.203:6789/0 2024-04-04T21:01:50.262 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.259+0000 7fd67fd6a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6780b1d60 addr v1:172.21.15.106:6789/0 2024-04-04T21:01:50.262 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.259+0000 7fd67fd6a640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6780aa2e0 0x7fd67813fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:50.262 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.260+0000 7fd67fd6a640 10 monclient(hunting): picked mon.noname-b con 0x7fd6780aa2e0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:50.262 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.260+0000 7fd67fd6a640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd678142cb0 con 0x7fd6780b1d60 2024-04-04T21:01:50.262 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.260+0000 7fd67fd6a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6780ab3d0 con 0x7fd6780b51a0 2024-04-04T21:01:50.262 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.260+0000 7fd67fd6a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:50.262 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.260+0000 7fd67fd6a640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6780b5580 con 0x7fd6780aa2e0 2024-04-04T21:01:50.262 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.260+0000 7fd67fd6a640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:50.262 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.260+0000 7fd67fd6a640 10 monclient(hunting): authenticate will time out at 1359.527365s 2024-04-04T21:01:50.262 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.260+0000 7fd67db06640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd6780b51a0 0x7fd6781425a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:37662/0 (socket says 172.21.15.106:37662) 2024-04-04T21:01:50.262 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.260+0000 7fd67db06640 1 -- 172.21.15.106:0/147606867 learned_addr learned my addr 172.21.15.106:0/147606867 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:01:50.263 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.260+0000 7fd67e307640 1 --2- 172.21.15.106:0/147606867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6780aa2e0 0x7fd67813fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:50.263 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.261+0000 7fd67e307640 10 monclient(hunting): get_auth_request con 0x7fd6780aa2e0 auth_method 0 2024-04-04T21:01:50.263 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.261+0000 7fd67e307640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:50.263 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.261+0000 7fd67e307640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:50.263 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.261+0000 7fd67e307640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:50.263 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.261+0000 7fd67e307640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:50.263 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.261+0000 7fd67e307640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:50.263 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.261+0000 7fd67e307640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:50.264 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.263+0000 7fd67cb04640 1 -- 172.21.15.106:0/147606867 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204148579 0 0) 0x7fd678142cb0 con 0x7fd6780b1d60 2024-04-04T21:01:50.264 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.263+0000 7fd67cb04640 10 monclient(hunting): _negotiate 2024-04-04T21:01:50.264 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.263+0000 7fd67cb04640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:50.264 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.263+0000 7fd67cb04640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:50.264 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.263+0000 7fd67cb04640 10 monclient(hunting): my global_id is 4200 2024-04-04T21:01:50.264 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.263+0000 7fd67cb04640 1 -- 172.21.15.106:0/147606867 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd664003520 con 0x7fd6780b1d60 2024-04-04T21:01:50.265 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.263+0000 7fd67e307640 10 monclient(hunting): handle_auth_done global_id 4184 payload 274 2024-04-04T21:01:50.265 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.263+0000 7fd67e307640 1 -- 172.21.15.106:0/147606867 >> v1:172.21.15.203:6789/0 conn(0x7fd6780b51a0 legacy=0x7fd6781425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:50.266 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.263+0000 7fd67e307640 1 -- 172.21.15.106:0/147606867 >> v1:172.21.15.106:6789/0 conn(0x7fd6780b1d60 legacy=0x7fd67813f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:50.266 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.263+0000 7fd67e307640 10 monclient: _finish_hunting 0 2024-04-04T21:01:50.266 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.263+0000 7fd67e307640 1 monclient: found mon.noname-b 2024-04-04T21:01:50.266 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.263+0000 7fd67e307640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:50.266 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.263+0000 7fd67e307640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:01:50.266 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.263+0000 7fd67e307640 1 -- 172.21.15.106:0/147606867 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6780a9d00 con 0x7fd6780aa2e0 2024-04-04T21:01:50.266 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.266+0000 7fd67e307640 1 --2- 172.21.15.106:0/147606867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6780aa2e0 0x7fd67813fe60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd670009dc0 tx=0x7fd67002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d5948522bd17f1b server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:50.279 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67cb04640 1 -- 172.21.15.106:0/147606867 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd670039070 con 0x7fd6780aa2e0 2024-04-04T21:01:50.279 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67cb04640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:50.279 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67cb04640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:50.279 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67cb04640 10 monclient: dump: 2024-04-04T21:01:50.279 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67cb04640 10 monclient: _finish_auth 0 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67cb04640 10 monclient: _check_auth_tickets 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67cb04640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67cb04640 1 -- 172.21.15.106:0/147606867 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67002cc50 con 0x7fd6780aa2e0 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67cb04640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:50.280 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67cb04640 1 -- 172.21.15.106:0/147606867 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd670034440 con 0x7fd6780aa2e0 2024-04-04T21:01:50.281 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67cb04640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:50.281 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67cb04640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:01:50.281 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67cb04640 10 monclient: dump: 2024-04-04T21:01:50.281 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:50.281 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:50.281 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:50.281 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:50.281 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:50.281 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:50.281 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:50.281 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:50.281 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:50.281 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:50.281 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67fd6a640 5 monclient: authenticate success, global_id 4184 2024-04-04T21:01:50.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67fd6a640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:50.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.271+0000 7fd67fd6a640 4 set_mon_vals no callback set 2024-04-04T21:01:50.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.272+0000 7fd67fd6a640 10 monclient: shutdown 2024-04-04T21:01:50.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.272+0000 7fd67fd6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:50.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.272+0000 7fd67fd6a640 1 -- 172.21.15.106:0/147606867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6780aa2e0 msgr2=0x7fd67813fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:50.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.272+0000 7fd67fd6a640 1 --2- 172.21.15.106:0/147606867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6780aa2e0 0x7fd67813fe60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd670009dc0 tx=0x7fd67002bea0 comp rx=0 tx=0).stop 2024-04-04T21:01:50.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.272+0000 7fd67fd6a640 1 -- 172.21.15.106:0/147606867 shutdown_connections 2024-04-04T21:01:50.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.272+0000 7fd67fd6a640 1 --2- 172.21.15.106:0/147606867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6780aa2e0 0x7fd67813fe60 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:50.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.272+0000 7fd67fd6a640 1 -- 172.21.15.106:0/147606867 >> 172.21.15.106:0/147606867 conn(0x7fd6780727a0 msgr2=0x7fd678072bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:50.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.272+0000 7fd67fd6a640 1 -- 172.21.15.106:0/147606867 shutdown_connections 2024-04-04T21:01:50.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.272+0000 7fd67fd6a640 1 -- 172.21.15.106:0/147606867 wait complete. 2024-04-04T21:01:50.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.272+0000 7fd67fd6a640 10 monclient: build_initial_monmap 2024-04-04T21:01:50.282 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.272+0000 7fd67fd6a640 10 monclient: monmap: 2024-04-04T21:01:50.282 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr:last_changed 0.000000 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr:created 0.000000 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.273+0000 7fd67fd6a640 1 librados: starting msgr at 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.273+0000 7fd67fd6a640 1 librados: starting objecter 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.273+0000 7fd67fd6a640 1 Processor -- start 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.273+0000 7fd67fd6a640 1 -- start start 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.273+0000 7fd67fd6a640 1 librados: setting wanted keys 2024-04-04T21:01:50.283 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.273+0000 7fd67fd6a640 1 librados: calling monclient init 2024-04-04T21:01:50.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.273+0000 7fd67fd6a640 10 monclient: init 2024-04-04T21:01:50.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:50.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-04T21:01:50.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6780aa2e0 0x7fd6780b7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:50.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 10 monclient(hunting): picked mon.noname-b con 0x7fd6780aa2e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:50.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6780b1d60 0x7fd6781df800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:50.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 10 monclient(hunting): picked mon.noname-c con 0x7fd6780b1d60 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:50.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6780b51a0 0x7fd678139090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:50.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 10 monclient(hunting): picked mon.noname-a con 0x7fd6780b51a0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:50.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:50.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67814e090 con 0x7fd6780b51a0 2024-04-04T21:01:50.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:50.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6780a9870 con 0x7fd6780aa2e0 2024-04-04T21:01:50.284 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67814e210 con 0x7fd6780b1d60 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67fd6a640 10 monclient(hunting): authenticate will time out at 1359.541508s 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.274+0000 7fd67d305640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6780b1d60 0x7fd6781df800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67db06640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6780aa2e0 0x7fd6780b7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67e307640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6780b51a0 0x7fd678139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67d305640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6780b1d60 0x7fd6781df800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:52974/0 (socket says 172.21.15.106:52974) 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67d305640 1 -- 172.21.15.106:0/1019765328 learned_addr learned my addr 172.21.15.106:0/1019765328 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67d305640 10 monclient(hunting): get_auth_request con 0x7fd6780b1d60 auth_method 0 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67d305640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67d305640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67d305640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67db06640 10 monclient(hunting): get_auth_request con 0x7fd6780aa2e0 auth_method 0 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67db06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:50.285 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67db06640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:50.286 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67db06640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:50.286 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67e307640 10 monclient(hunting): get_auth_request con 0x7fd6780b51a0 auth_method 0 2024-04-04T21:01:50.286 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67e307640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:50.286 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67e307640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:50.286 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67e307640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:50.286 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67d305640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:50.286 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67d305640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:50.286 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.275+0000 7fd67d305640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:50.286 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.276+0000 7fd67db06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.276+0000 7fd67db06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.276+0000 7fd67db06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.276+0000 7fd67e307640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.276+0000 7fd67e307640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.276+0000 7fd67e307640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.278+0000 7fd67d305640 10 monclient(hunting): handle_auth_done global_id 4187 payload 824 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.278+0000 7fd67d305640 1 -- 172.21.15.106:0/1019765328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6780aa2e0 msgr2=0x7fd6780b7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.278+0000 7fd67d305640 1 --2- 172.21.15.106:0/1019765328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6780aa2e0 0x7fd6780b7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.278+0000 7fd67d305640 1 -- 172.21.15.106:0/1019765328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6780b51a0 msgr2=0x7fd678139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.278+0000 7fd67d305640 1 --2- 172.21.15.106:0/1019765328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6780b51a0 0x7fd678139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.278+0000 7fd67d305640 10 monclient: _finish_hunting 0 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.278+0000 7fd67d305640 1 monclient: found mon.noname-c 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.278+0000 7fd67d305640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.278+0000 7fd67d305640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:01:50.287 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.278+0000 7fd67d305640 1 -- 172.21.15.106:0/1019765328 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6781dfd40 con 0x7fd6780b1d60 2024-04-04T21:01:50.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.281+0000 7fd67d305640 1 --2- 172.21.15.106:0/1019765328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6780b1d60 0x7fd6781df800 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd66800c490 tx=0x7fd66800c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:50.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.281+0000 7fd66e7fc640 10 client.?.objecter ms_handle_connect 0x7fd6780b1d60 2024-04-04T21:01:50.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.281+0000 7fd66e7fc640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:50.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.284+0000 7fd66e7fc640 1 -- 172.21.15.106:0/1019765328 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6680052d0 con 0x7fd6780b1d60 2024-04-04T21:01:50.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.284+0000 7fd66e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:50.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.284+0000 7fd66e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:01:50.291 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.284+0000 7fd66e7fc640 10 monclient: dump: 2024-04-04T21:01:50.291 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:50.291 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:50.291 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:50.291 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:50.291 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.285+0000 7fd66e7fc640 10 monclient: _finish_auth 0 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.285+0000 7fd66e7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.285+0000 7fd66e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.285+0000 7fd66e7fc640 1 -- 172.21.15.106:0/1019765328 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd668005470 con 0x7fd6780b1d60 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.285+0000 7fd66e7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.285+0000 7fd66ffff640 4 set_mon_vals no callback set 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.285+0000 7fd66e7fc640 1 -- 172.21.15.106:0/1019765328 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd668012850 con 0x7fd6780b1d60 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.285+0000 7fd67fd6a640 5 monclient: authenticate success, global_id 4187 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.285+0000 7fd67fd6a640 10 monclient: _renew_subs 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.285+0000 7fd67fd6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:50.292 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.285+0000 7fd67fd6a640 1 -- 172.21.15.106:0/1019765328 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd678071e90 con 0x7fd6780b1d60 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.286+0000 7fd66e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.286+0000 7fd66e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.286+0000 7fd66e7fc640 10 monclient: dump: 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.286+0000 7fd67fd6a640 10 client.4187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:50.293 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.286+0000 7fd67fd6a640 10 monclient: _renew_subs 2024-04-04T21:01:50.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.286+0000 7fd67fd6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:50.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.286+0000 7fd67fd6a640 1 -- 172.21.15.106:0/1019765328 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6781397b0 con 0x7fd6780b1d60 2024-04-04T21:01:50.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.286+0000 7fd67fd6a640 1 librados: init done 2024-04-04T21:01:50.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.287+0000 7fd67fd6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:50.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.287+0000 7fd67fd6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:50.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.287+0000 7fd67fd6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:50.294 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.287+0000 7fd67fd6a640 1 -- 172.21.15.106:0/1019765328 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd67806a560 con 0x7fd6780b1d60 2024-04-04T21:01:50.309 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.308+0000 7fd66e7fc640 1 -- 172.21.15.106:0/1019765328 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fd6680129f0 con 0x7fd6780b1d60 2024-04-04T21:01:50.310 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.308+0000 7fd66e7fc640 1 --2- 172.21.15.106:0/1019765328 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd65c06ce20 0x7fd65c06f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:50.310 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.309+0000 7fd67db06640 1 --2- 172.21.15.106:0/1019765328 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd65c06ce20 0x7fd65c06f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:50.310 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.309+0000 7fd67db06640 10 monclient: get_auth_request con 0x7fd65c06ce20 auth_method 0 2024-04-04T21:01:50.310 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.310+0000 7fd67db06640 1 --2- 172.21.15.106:0/1019765328 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd65c06ce20 0x7fd65c06f2e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd674005880 tx=0x7fd674009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:50.310 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.310+0000 7fd66e7fc640 10 client.4187.objecter ms_handle_connect 0x7fd65c06ce20 2024-04-04T21:01:50.416 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.416+0000 7fd66e7fc640 1 -- 172.21.15.106:0/1019765328 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fd668057d00 con 0x7fd6780b1d60 2024-04-04T21:01:50.417 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.416+0000 7fd66e7fc640 10 client.4187.objecter ms_dispatch 0x7fd678000b90 osd_map(10..10 src has 1..10) v4 2024-04-04T21:01:50.417 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.416+0000 7fd66e7fc640 3 client.4187.objecter handle_osd_map got epochs [10,10] > 0 2024-04-04T21:01:50.417 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.416+0000 7fd66e7fc640 3 client.4187.objecter handle_osd_map decoding full epoch 10 2024-04-04T21:01:50.417 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.416+0000 7fd66e7fc640 20 client.4187.objecter dump_active .. 0 homeless 2024-04-04T21:01:50.696 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.695+0000 7fd66e7fc640 1 -- 172.21.15.106:0/1019765328 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd668059070 con 0x7fd6780b1d60 2024-04-04T21:01:50.696 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.695+0000 7fd66e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:50.696 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.695+0000 7fd66e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:50.918 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.918+0000 7fd67fd6a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:50.919 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.918+0000 7fd67fd6a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:50.919 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.918+0000 7fd67fd6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:50.919 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.918+0000 7fd67fd6a640 1 -- 172.21.15.106:0/1019765328 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd67813e140 con 0x7fd6780b1d60 2024-04-04T21:01:50.934 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.934+0000 7fd66e7fc640 1 -- 172.21.15.106:0/1019765328 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fd66805ced0 con 0x7fd6780b1d60 2024-04-04T21:01:50.935 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.934+0000 7fd66e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:50.935 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.934+0000 7fd66e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:01:50.935 INFO:teuthology.misc.health.smithi106.stdout: 2024-04-04T21:01:50.935 INFO:teuthology.misc.health.smithi106.stdout:{"epoch":10,"fsid":"8f89ce43-1bcb-41cf-9555-c40eeb258155","created":"2024-04-04T21:01:20.057083+0000","modified":"2024-04-04T21:01:25.642687+0000","last_up_change":"0.000000","last_in_change":"2024-04-04T21:01:25.642687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"28337134-5253-454a-bcda-47267b05b2c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ae336a0e-2a90-4323-9f0f-1701fffe6b66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"773087e0-78c9-4d14-bf0a-fb605c21f92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07da9513-c746-46d1-995f-6f8e91338b84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c47d034c-1db6-4332-bf24-722fb2846808","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"588caa62-e773-44ae-94e5-ece54fe980d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ef776e12-7c80-4095-b173-5b2d24996f95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"53aafc58-6463-4aeb-8395-e14fcfcee5de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-04T21:01:50.938 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.936+0000 7fd67fd6a640 10 librados: watch_flush enter 2024-04-04T21:01:50.938 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.936+0000 7fd67fd6a640 10 librados: watch_flush exit 2024-04-04T21:01:50.939 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.936+0000 7fd67fd6a640 20 client.4187.objecter shutdown clearing up homeless session... 2024-04-04T21:01:50.939 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.936+0000 7fd67fd6a640 10 client.4187.objecter successfully canceled tick 2024-04-04T21:01:50.939 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.936+0000 7fd67fd6a640 1 -- 172.21.15.106:0/1019765328 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd65c06ce20 msgr2=0x7fd65c06f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:50.939 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.936+0000 7fd67fd6a640 1 --2- 172.21.15.106:0/1019765328 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd65c06ce20 0x7fd65c06f2e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd674005880 tx=0x7fd674009040 comp rx=0 tx=0).stop 2024-04-04T21:01:50.939 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.937+0000 7fd67fd6a640 10 monclient: shutdown 2024-04-04T21:01:50.939 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.937+0000 7fd67fd6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:50.939 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.937+0000 7fd67fd6a640 1 -- 172.21.15.106:0/1019765328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6780b1d60 msgr2=0x7fd6781df800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:50.939 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.937+0000 7fd67fd6a640 1 --2- 172.21.15.106:0/1019765328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6780b1d60 0x7fd6781df800 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd66800c490 tx=0x7fd66800c950 comp rx=0 tx=0).stop 2024-04-04T21:01:50.939 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.937+0000 7fd67fd6a640 1 -- 172.21.15.106:0/1019765328 shutdown_connections 2024-04-04T21:01:50.939 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.937+0000 7fd67fd6a640 1 --2- 172.21.15.106:0/1019765328 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd65c06ce20 0x7fd65c06f2e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:50.939 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.937+0000 7fd67fd6a640 1 --2- 172.21.15.106:0/1019765328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6780b51a0 0x7fd678139090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:50.939 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.937+0000 7fd67fd6a640 1 --2- 172.21.15.106:0/1019765328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6780b1d60 0x7fd6781df800 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd66800c490 tx=0x7fd66800c950 comp rx=0 tx=0).stop 2024-04-04T21:01:50.939 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.937+0000 7fd67fd6a640 1 --2- 172.21.15.106:0/1019765328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6780aa2e0 0x7fd6780b7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:50.939 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.937+0000 7fd67fd6a640 1 -- 172.21.15.106:0/1019765328 >> 172.21.15.106:0/1019765328 conn(0x7fd6780727a0 msgr2=0x7fd67813f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:50.940 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.937+0000 7fd67fd6a640 1 -- 172.21.15.106:0/1019765328 shutdown_connections 2024-04-04T21:01:50.940 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.937+0000 7fd67fd6a640 1 -- 172.21.15.106:0/1019765328 wait complete. 2024-04-04T21:01:50.940 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:50.937+0000 7fd67fd6a640 1 librados: shutdown 2024-04-04T21:01:50.949 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-04T21:01:51.386 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:25.570 35364== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:51.674 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:25.876 35362== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:51.997 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:26.190 35363== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:52.465 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:26.671 35361== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:53.010 INFO:tasks.ceph.osd.0.smithi106.stderr:==00:00:00:27.210 35501== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:53.069 INFO:tasks.ceph.osd.2.smithi106.stderr:==00:00:00:27.287 35499== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:53.638 INFO:tasks.ceph.osd.1.smithi106.stderr:==00:00:00:27.838 35502== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:53.903 INFO:tasks.ceph.osd.3.smithi106.stderr:==00:00:00:28.117 35500== Warning: unimplemented fcntl command: 1038 2024-04-04T21:01:57.756 DEBUG:teuthology.orchestra.run.smithi106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-04T21:01:57.848 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.847+0000 7f057198a640 10 monclient: get_monmap_and_config 2024-04-04T21:01:57.848 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.847+0000 7f057198a640 10 monclient: build_initial_monmap 2024-04-04T21:01:57.848 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.847+0000 7f057198a640 10 monclient: monmap: 2024-04-04T21:01:57.848 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:01:57.848 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:57.848 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:01:57.848260+0000 2024-04-04T21:01:57.848 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:01:57.848260+0000 2024-04-04T21:01:57.848 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:57.848 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:57.848 INFO:teuthology.misc.health.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:01:57.849 INFO:teuthology.misc.health.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:01:57.849 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:01:57.849 INFO:teuthology.misc.health.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:01:57.849 INFO:teuthology.misc.health.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:01:57.849 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:57.851 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.848+0000 7f057198a640 1 Processor -- start 2024-04-04T21:01:57.851 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.848+0000 7f057198a640 1 -- start start 2024-04-04T21:01:57.851 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.848+0000 7f057198a640 10 monclient: init 2024-04-04T21:01:57.851 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.849+0000 7f057198a640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:57.851 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.849+0000 7f057198a640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-04-04T21:01:57.851 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.849+0000 7f057198a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f056c0b87e0 addr v1:172.21.15.203:6789/0 2024-04-04T21:01:57.851 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.849+0000 7f057198a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f056c0aa300 addr v1:172.21.15.106:6789/0 2024-04-04T21:01:57.851 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.849+0000 7f057198a640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f056c13fc70 0x7f056c14df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:57.851 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.849+0000 7f057198a640 10 monclient(hunting): picked mon.noname-b con 0x7f056c13fc70 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:57.852 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.849+0000 7f057198a640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056c0ab360 con 0x7f056c0aa300 2024-04-04T21:01:57.852 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.849+0000 7f057198a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056c0ab640 con 0x7f056c0b87e0 2024-04-04T21:01:57.852 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.849+0000 7f057198a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:57.852 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.849+0000 7f057198a640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0b8bc0 con 0x7f056c13fc70 2024-04-04T21:01:57.852 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.849+0000 7f057198a640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:57.852 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.849+0000 7f057198a640 10 monclient(hunting): authenticate will time out at 1367.117171s 2024-04-04T21:01:57.852 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.850+0000 7f0563fff640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f056c0aa300 0x7f056c13f560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:55706/0 (socket says 172.21.15.106:55706) 2024-04-04T21:01:57.852 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.850+0000 7f0563fff640 1 -- 172.21.15.106:0/3526541750 learned_addr learned my addr 172.21.15.106:0/3526541750 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:01:57.852 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.850+0000 7f056b7fe640 1 --2- 172.21.15.106:0/3526541750 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f056c13fc70 0x7f056c14df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:57.853 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.851+0000 7f056b7fe640 10 monclient(hunting): get_auth_request con 0x7f056c13fc70 auth_method 0 2024-04-04T21:01:57.853 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.851+0000 7f056b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:57.853 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.851+0000 7f056b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:57.853 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.851+0000 7f056b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:57.853 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.852+0000 7f056b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:57.853 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.852+0000 7f056b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:57.853 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.852+0000 7f056b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:57.855 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.854+0000 7f056b7fe640 10 monclient(hunting): handle_auth_done global_id 4190 payload 274 2024-04-04T21:01:57.855 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.854+0000 7f056b7fe640 1 -- 172.21.15.106:0/3526541750 >> v1:172.21.15.203:6789/0 conn(0x7f056c0b87e0 legacy=0x7f056c142650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:57.855 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.854+0000 7f056b7fe640 1 -- 172.21.15.106:0/3526541750 >> v1:172.21.15.106:6789/0 conn(0x7f056c0aa300 legacy=0x7f056c13f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:57.855 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.854+0000 7f056b7fe640 10 monclient: _finish_hunting 0 2024-04-04T21:01:57.855 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.854+0000 7f056b7fe640 1 monclient: found mon.noname-b 2024-04-04T21:01:57.855 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.854+0000 7f056b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:57.855 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.854+0000 7f056b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:01:57.855 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.854+0000 7f056b7fe640 1 -- 172.21.15.106:0/3526541750 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c140190 con 0x7f056c13fc70 2024-04-04T21:01:57.856 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.856+0000 7f056b7fe640 1 --2- 172.21.15.106:0/3526541750 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f056c13fc70 0x7f056c14df70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0564009dc0 tx=0x7f056402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28848e9e0b4cc088 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:57.863 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f056a7fc640 1 -- 172.21.15.106:0/3526541750 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564039070 con 0x7f056c13fc70 2024-04-04T21:01:57.863 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f056a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:57.863 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f056a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:01:57.863 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f056a7fc640 10 monclient: dump: 2024-04-04T21:01:57.863 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:57.863 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:57.863 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:57.863 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:57.863 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:57.863 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:57.863 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:57.863 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:57.863 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f056a7fc640 10 monclient: _finish_auth 0 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f056a7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f056a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f056a7fc640 1 -- 172.21.15.106:0/3526541750 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056402cc40 con 0x7f056c13fc70 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f056a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f057198a640 5 monclient: authenticate success, global_id 4190 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f056a7fc640 1 -- 172.21.15.106:0/3526541750 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564034440 con 0x7f056c13fc70 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f057198a640 10 monclient: get_monmap_and_config success 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f057198a640 4 set_mon_vals no callback set 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f056a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f056a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.860+0000 7f056a7fc640 10 monclient: dump: 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:57.864 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.861+0000 7f057198a640 10 monclient: shutdown 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.861+0000 7f057198a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.861+0000 7f057198a640 1 -- 172.21.15.106:0/3526541750 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f056c13fc70 msgr2=0x7f056c14df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.861+0000 7f057198a640 1 --2- 172.21.15.106:0/3526541750 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f056c13fc70 0x7f056c14df70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0564009dc0 tx=0x7f056402be90 comp rx=0 tx=0).stop 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.861+0000 7f057198a640 1 -- 172.21.15.106:0/3526541750 shutdown_connections 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.861+0000 7f057198a640 1 --2- 172.21.15.106:0/3526541750 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f056c13fc70 0x7f056c14df70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:57.865 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.861+0000 7f057198a640 1 -- 172.21.15.106:0/3526541750 >> 172.21.15.106:0/3526541750 conn(0x7f056c0727d0 msgr2=0x7f056c072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:57.866 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.861+0000 7f057198a640 1 -- 172.21.15.106:0/3526541750 shutdown_connections 2024-04-04T21:01:57.866 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.861+0000 7f057198a640 1 -- 172.21.15.106:0/3526541750 wait complete. 2024-04-04T21:01:57.869 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.863+0000 7f057198a640 10 monclient: build_initial_monmap 2024-04-04T21:01:57.869 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.863+0000 7f057198a640 10 monclient: monmap: 2024-04-04T21:01:57.869 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:01:57.869 INFO:teuthology.misc.health.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:01:57.869 INFO:teuthology.misc.health.smithi106.stderr:last_changed 0.000000 2024-04-04T21:01:57.869 INFO:teuthology.misc.health.smithi106.stderr:created 0.000000 2024-04-04T21:01:57.870 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:01:57.870 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:57.870 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:01:57.870 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:01:57.870 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:01:57.870 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:57.870 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.863+0000 7f057198a640 1 librados: starting msgr at 2024-04-04T21:01:57.870 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.863+0000 7f057198a640 1 librados: starting objecter 2024-04-04T21:01:57.870 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.863+0000 7f057198a640 1 Processor -- start 2024-04-04T21:01:57.870 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.863+0000 7f057198a640 1 -- start start 2024-04-04T21:01:57.870 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.863+0000 7f057198a640 1 librados: setting wanted keys 2024-04-04T21:01:57.870 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.863+0000 7f057198a640 1 librados: calling monclient init 2024-04-04T21:01:57.870 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.863+0000 7f057198a640 10 monclient: init 2024-04-04T21:01:57.871 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 10 monclient: _reopen_session rank -1 2024-04-04T21:01:57.871 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-04T21:01:57.871 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f056c0aa300 0x7f056c071710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:57.871 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 10 monclient(hunting): picked mon.noname-b con 0x7f056c0aa300 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:01:57.871 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f056c0b87e0 0x7f056c071c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:57.871 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 10 monclient(hunting): picked mon.noname-a con 0x7f056c0b87e0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:01:57.871 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f056c13fc70 0x7f056c1dfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:57.871 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 10 monclient(hunting): picked mon.noname-c con 0x7f056c13fc70 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:01:57.871 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:57.871 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0a9b80 con 0x7f056c0b87e0 2024-04-04T21:01:57.871 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:57.871 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c0a9a00 con 0x7f056c0aa300 2024-04-04T21:01:57.871 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:01:57.871 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c142d60 con 0x7f056c13fc70 2024-04-04T21:01:57.872 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 10 monclient(hunting): _renew_subs 2024-04-04T21:01:57.872 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.864+0000 7f057198a640 10 monclient(hunting): authenticate will time out at 1367.132073s 2024-04-04T21:01:57.872 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.865+0000 7f056b7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f056c13fc70 0x7f056c1dfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:57.872 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.865+0000 7f056affd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f056c0aa300 0x7f056c071710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:57.872 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.865+0000 7f056affd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f056c0aa300 0x7f056c071710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:46168/0 (socket says 172.21.15.106:46168) 2024-04-04T21:01:57.872 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.865+0000 7f056b7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f056c13fc70 0x7f056c1dfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:53436/0 (socket says 172.21.15.106:53436) 2024-04-04T21:01:57.872 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.865+0000 7f056affd640 1 -- 172.21.15.106:0/1663587200 learned_addr learned my addr 172.21.15.106:0/1663587200 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:01:57.872 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.865+0000 7f056affd640 10 monclient(hunting): get_auth_request con 0x7f056c0aa300 auth_method 0 2024-04-04T21:01:57.872 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.865+0000 7f056affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:57.872 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.865+0000 7f056affd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:57.872 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.865+0000 7f056affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:57.872 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.865+0000 7f056b7fe640 10 monclient(hunting): get_auth_request con 0x7f056c13fc70 auth_method 0 2024-04-04T21:01:57.872 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.865+0000 7f056b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:01:57.872 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.865+0000 7f056b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:01:57.873 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.865+0000 7f056b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:01:57.873 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.866+0000 7f056affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:57.873 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.866+0000 7f056affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:57.873 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.866+0000 7f056affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:57.873 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.866+0000 7f056b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:01:57.873 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.866+0000 7f056b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:01:57.873 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.866+0000 7f056b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:01:57.874 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.870+0000 7f056b7fe640 10 monclient(hunting): handle_auth_done global_id 4193 payload 824 2024-04-04T21:01:57.875 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.870+0000 7f056b7fe640 1 -- 172.21.15.106:0/1663587200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f056c0aa300 msgr2=0x7f056c071710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:57.875 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.870+0000 7f056b7fe640 1 --2- 172.21.15.106:0/1663587200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f056c0aa300 0x7f056c071710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:57.875 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.870+0000 7f056b7fe640 1 -- 172.21.15.106:0/1663587200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f056c0b87e0 msgr2=0x7f056c071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:57.875 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.870+0000 7f056b7fe640 1 --2- 172.21.15.106:0/1663587200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f056c0b87e0 0x7f056c071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:57.875 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.870+0000 7f056b7fe640 10 monclient: _finish_hunting 0 2024-04-04T21:01:57.875 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.870+0000 7f056b7fe640 1 monclient: found mon.noname-c 2024-04-04T21:01:57.875 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.870+0000 7f056b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:01:57.875 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.870+0000 7f056b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:01:57.875 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.870+0000 7f056b7fe640 1 -- 172.21.15.106:0/1663587200 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c072190 con 0x7f056c13fc70 2024-04-04T21:01:57.875 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.870+0000 7f056affd640 1 --2- 172.21.15.106:0/1663587200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f056c0aa300 0x7f056c071710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-04T21:01:57.875 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.874+0000 7f056b7fe640 1 --2- 172.21.15.106:0/1663587200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f056c13fc70 0x7f056c1dfac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0564033000 tx=0x7f0564004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:57.875 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.874+0000 7f0568ff9640 10 client.?.objecter ms_handle_connect 0x7f056c13fc70 2024-04-04T21:01:57.875 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.874+0000 7f0568ff9640 10 client.?.objecter resend_mon_ops 2024-04-04T21:01:57.955 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f0568ff9640 1 -- 172.21.15.106:0/1663587200 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564044020 con 0x7f056c13fc70 2024-04-04T21:01:57.955 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f0568ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:57.955 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f0568ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:01:57.955 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f0568ff9640 10 monclient: dump: 2024-04-04T21:01:57.955 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:57.955 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:57.956 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:57.956 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:57.956 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:57.956 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:57.956 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:57.956 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:57.956 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:57.956 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:57.956 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f0568ff9640 10 monclient: _finish_auth 0 2024-04-04T21:01:57.956 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f0568ff9640 10 monclient: _check_auth_tickets 2024-04-04T21:01:57.956 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f0568ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:01:57.956 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f0568ff9640 1 -- 172.21.15.106:0/1663587200 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056403f470 con 0x7f056c13fc70 2024-04-04T21:01:57.956 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f057198a640 5 monclient: authenticate success, global_id 4193 2024-04-04T21:01:57.956 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f057198a640 10 monclient: _renew_subs 2024-04-04T21:01:57.957 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f057198a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:57.957 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f057198a640 1 -- 172.21.15.106:0/1663587200 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056c06ffb0 con 0x7f056c13fc70 2024-04-04T21:01:57.957 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f0568ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:01:57.957 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f0569ffb640 4 set_mon_vals no callback set 2024-04-04T21:01:57.958 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.954+0000 7f057198a640 10 client.4193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:01:57.958 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.955+0000 7f0568ff9640 1 -- 172.21.15.106:0/1663587200 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564039040 con 0x7f056c13fc70 2024-04-04T21:01:57.958 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.955+0000 7f0568ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:01:57.958 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.955+0000 7f0568ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:01:57.958 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.955+0000 7f0568ff9640 10 monclient: dump: 2024-04-04T21:01:57.958 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:01:57.958 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:01:57.958 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:57.958 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:01:57.958 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:01:57.958 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:01:57.958 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:01:57.958 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:01:57.959 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:01:57.959 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:01:57.959 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.955+0000 7f057198a640 10 monclient: _renew_subs 2024-04-04T21:01:57.959 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.955+0000 7f057198a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:57.959 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.955+0000 7f057198a640 1 -- 172.21.15.106:0/1663587200 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056c06dfa0 con 0x7f056c13fc70 2024-04-04T21:01:57.959 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.955+0000 7f057198a640 1 librados: init done 2024-04-04T21:01:57.959 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.956+0000 7f057198a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:57.959 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.956+0000 7f057198a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:57.959 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.956+0000 7f057198a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:57.959 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:57.956+0000 7f057198a640 1 -- 172.21.15.106:0/1663587200 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056c06e230 con 0x7f056c13fc70 2024-04-04T21:01:58.214 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.213+0000 7f0568ff9640 1 -- 172.21.15.106:0/1663587200 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f0564047020 con 0x7f056c13fc70 2024-04-04T21:01:58.214 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.213+0000 7f0568ff9640 1 --2- 172.21.15.106:0/1663587200 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f054c06ce00 0x7f054c06f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:01:58.214 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.213+0000 7f056affd640 1 --2- 172.21.15.106:0/1663587200 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f054c06ce00 0x7f054c06f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:01:58.214 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.214+0000 7f056affd640 10 monclient: get_auth_request con 0x7f054c06ce00 auth_method 0 2024-04-04T21:01:58.215 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.214+0000 7f056affd640 1 --2- 172.21.15.106:0/1663587200 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f054c06ce00 0x7f054c06f2c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f055c007fd0 tx=0x7f055c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:01:58.215 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.214+0000 7f0568ff9640 10 client.4193.objecter ms_handle_connect 0x7f054c06ce00 2024-04-04T21:01:58.229 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.228+0000 7f0568ff9640 1 -- 172.21.15.106:0/1663587200 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f0564079660 con 0x7f056c13fc70 2024-04-04T21:01:58.229 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.228+0000 7f0568ff9640 10 client.4193.objecter ms_dispatch 0x7f056c000b90 osd_map(10..10 src has 1..10) v4 2024-04-04T21:01:58.229 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.228+0000 7f0568ff9640 3 client.4193.objecter handle_osd_map got epochs [10,10] > 0 2024-04-04T21:01:58.229 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.228+0000 7f0568ff9640 3 client.4193.objecter handle_osd_map decoding full epoch 10 2024-04-04T21:01:58.229 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.228+0000 7f0568ff9640 20 client.4193.objecter dump_active .. 0 homeless 2024-04-04T21:01:58.229 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.228+0000 7f0568ff9640 1 -- 172.21.15.106:0/1663587200 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f056404e030 con 0x7f056c13fc70 2024-04-04T21:01:58.229 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.228+0000 7f0568ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:01:58.230 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.228+0000 7f0568ff9640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:01:58.416 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.415+0000 7f057198a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:58.416 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.415+0000 7f057198a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:58.416 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.415+0000 7f057198a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:01:58.416 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.415+0000 7f057198a640 1 -- 172.21.15.106:0/1663587200 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f056c06dd50 con 0x7f056c13fc70 2024-04-04T21:01:58.442 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.441+0000 7f0568ff9640 1 -- 172.21.15.106:0/1663587200 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f056407bcc0 con 0x7f056c13fc70 2024-04-04T21:01:58.442 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.441+0000 7f0568ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:01:58.442 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.441+0000 7f0568ff9640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:01:58.442 INFO:teuthology.misc.health.smithi106.stdout: 2024-04-04T21:01:58.442 INFO:teuthology.misc.health.smithi106.stdout:{"epoch":10,"fsid":"8f89ce43-1bcb-41cf-9555-c40eeb258155","created":"2024-04-04T21:01:20.057083+0000","modified":"2024-04-04T21:01:25.642687+0000","last_up_change":"0.000000","last_in_change":"2024-04-04T21:01:25.642687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"28337134-5253-454a-bcda-47267b05b2c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ae336a0e-2a90-4323-9f0f-1701fffe6b66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"773087e0-78c9-4d14-bf0a-fb605c21f92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07da9513-c746-46d1-995f-6f8e91338b84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c47d034c-1db6-4332-bf24-722fb2846808","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"588caa62-e773-44ae-94e5-ece54fe980d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ef776e12-7c80-4095-b173-5b2d24996f95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"53aafc58-6463-4aeb-8395-e14fcfcee5de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-04T21:01:58.445 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 10 librados: watch_flush enter 2024-04-04T21:01:58.445 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 10 librados: watch_flush exit 2024-04-04T21:01:58.445 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 20 client.4193.objecter shutdown clearing up homeless session... 2024-04-04T21:01:58.445 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 10 client.4193.objecter successfully canceled tick 2024-04-04T21:01:58.445 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 1 -- 172.21.15.106:0/1663587200 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f054c06ce00 msgr2=0x7f054c06f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:58.445 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 1 --2- 172.21.15.106:0/1663587200 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f054c06ce00 0x7f054c06f2c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f055c007fd0 tx=0x7f055c009040 comp rx=0 tx=0).stop 2024-04-04T21:01:58.445 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 10 monclient: shutdown 2024-04-04T21:01:58.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:01:58.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 1 -- 172.21.15.106:0/1663587200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f056c13fc70 msgr2=0x7f056c1dfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:01:58.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 1 --2- 172.21.15.106:0/1663587200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f056c13fc70 0x7f056c1dfac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0564033000 tx=0x7f0564004a40 comp rx=0 tx=0).stop 2024-04-04T21:01:58.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 1 -- 172.21.15.106:0/1663587200 shutdown_connections 2024-04-04T21:01:58.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 1 --2- 172.21.15.106:0/1663587200 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f054c06ce00 0x7f054c06f2c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:58.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 1 --2- 172.21.15.106:0/1663587200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f056c13fc70 0x7f056c1dfac0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:58.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 1 --2- 172.21.15.106:0/1663587200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f056c0b87e0 0x7f056c071c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:58.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 1 --2- 172.21.15.106:0/1663587200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f056c0aa300 0x7f056c071710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:01:58.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.443+0000 7f057198a640 1 -- 172.21.15.106:0/1663587200 >> 172.21.15.106:0/1663587200 conn(0x7f056c0727d0 msgr2=0x7f056c0b64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:01:58.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.444+0000 7f057198a640 1 -- 172.21.15.106:0/1663587200 shutdown_connections 2024-04-04T21:01:58.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.444+0000 7f057198a640 1 -- 172.21.15.106:0/1663587200 wait complete. 2024-04-04T21:01:58.446 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:01:58.444+0000 7f057198a640 1 librados: shutdown 2024-04-04T21:01:58.458 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-04T21:01:59.621 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-04T21:01:59.618+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-04-04T21:01:59.919 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:01:59.917+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-04-04T21:02:00.331 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-04-04T21:02:00.328+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-04-04T21:02:00.578 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:02:00.574+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-04-04T21:02:01.994 INFO:tasks.ceph.osd.0.smithi106.stderr:2024-04-04T21:02:01.990+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-04-04T21:02:02.177 INFO:tasks.ceph.osd.2.smithi106.stderr:2024-04-04T21:02:02.173+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-04-04T21:02:02.479 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-04T21:02:02.474+0000 c48f640 -1 osd.5 0 waiting for initial osdmap 2024-04-04T21:02:02.869 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-04T21:02:02.855+0000 10255640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-04T21:02:03.072 INFO:tasks.ceph.osd.3.smithi106.stderr:2024-04-04T21:02:03.069+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-04-04T21:02:03.140 INFO:tasks.ceph.osd.1.smithi106.stderr:2024-04-04T21:02:03.088+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-04-04T21:02:03.715 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-04-04T21:02:03.711+0000 c48f640 -1 osd.6 0 waiting for initial osdmap 2024-04-04T21:02:03.722 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:02:03.718+0000 c48f640 -1 osd.7 0 waiting for initial osdmap 2024-04-04T21:02:04.161 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-04-04T21:02:04.140+0000 10255640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-04T21:02:04.163 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:02:04.150+0000 10255640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-04T21:02:04.750 INFO:tasks.ceph.osd.0.smithi106.stderr:2024-04-04T21:02:04.747+0000 c48f640 -1 osd.0 0 waiting for initial osdmap 2024-04-04T21:02:05.159 INFO:tasks.ceph.osd.0.smithi106.stderr:2024-04-04T21:02:05.145+0000 10255640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-04T21:02:05.264 DEBUG:teuthology.orchestra.run.smithi106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-04T21:02:05.356 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.353+0000 7f15bc245640 10 monclient: get_monmap_and_config 2024-04-04T21:02:05.356 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.353+0000 7f15bc245640 10 monclient: build_initial_monmap 2024-04-04T21:02:05.356 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.353+0000 7f15bc245640 10 monclient: monmap: 2024-04-04T21:02:05.356 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:02:05.356 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:05.356 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:02:05.354641+0000 2024-04-04T21:02:05.356 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:02:05.354641+0000 2024-04-04T21:02:05.356 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:05.356 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:05.356 INFO:teuthology.misc.health.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:05.356 INFO:teuthology.misc.health.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:05.356 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:05.356 INFO:teuthology.misc.health.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.354+0000 7f15bc245640 1 Processor -- start 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.354+0000 7f15bc245640 1 -- start start 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.354+0000 7f15bc245640 10 monclient: init 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.354+0000 7f15bc245640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.354+0000 7f15bc245640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.354+0000 7f15bc245640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f15b406acc0 0x7f15b41403a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.354+0000 7f15bc245640 10 monclient(hunting): picked mon.noname-a con 0x7f15b406acc0 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.354+0000 7f15bc245640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15b406b0a0 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.355+0000 7f15bc245640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f15b40af6d0 0x7f15b40aed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.355+0000 7f15bc245640 10 monclient(hunting): picked mon.noname-c con 0x7f15b40af6d0 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.355+0000 7f15bc245640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b40b0550 con 0x7f15b406b0a0 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.355+0000 7f15bc245640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:05.357 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.355+0000 7f15bc245640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b40af240 con 0x7f15b406acc0 2024-04-04T21:02:05.358 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.355+0000 7f15bc245640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:05.358 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.355+0000 7f15bc245640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b40b1490 con 0x7f15b40af6d0 2024-04-04T21:02:05.358 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.355+0000 7f15bc245640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:05.358 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.355+0000 7f15bc245640 10 monclient(hunting): authenticate will time out at 1374.622241s 2024-04-04T21:02:05.358 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.355+0000 7f15b9fe1640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f15b406acc0 0x7f15b41403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:05.358 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.355+0000 7f15ba7e2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f15b40af6d0 0x7f15b40aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:05.359 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15ba7e2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f15b40af6d0 0x7f15b40aed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:46248/0 (socket says 172.21.15.106:46248) 2024-04-04T21:02:05.359 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15ba7e2640 1 -- 172.21.15.106:0/1114171746 learned_addr learned my addr 172.21.15.106:0/1114171746 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15ba7e2640 10 monclient(hunting): get_auth_request con 0x7f15b40af6d0 auth_method 0 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15ba7e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15ba7e2640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15ba7e2640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15b9fe1640 10 monclient(hunting): get_auth_request con 0x7f15b406acc0 auth_method 0 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15b9fe1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15b9fe1640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15b9fe1640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15ba7e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15ba7e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15ba7e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15b9fe1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15b9fe1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.356+0000 7f15b9fe1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:05.360 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.358+0000 7f15ba7e2640 10 monclient(hunting): handle_auth_done global_id 4255 payload 274 2024-04-04T21:02:05.361 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.358+0000 7f15ba7e2640 1 -- 172.21.15.106:0/1114171746 >> v2:172.21.15.106:3300/0 conn(0x7f15b406acc0 msgr2=0x7f15b41403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:05.361 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.358+0000 7f15ba7e2640 1 --2- 172.21.15.106:0/1114171746 >> v2:172.21.15.106:3300/0 conn(0x7f15b406acc0 0x7f15b41403a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:05.361 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.358+0000 7f15ba7e2640 1 -- 172.21.15.106:0/1114171746 >> v1:172.21.15.106:6789/0 conn(0x7f15b406b0a0 legacy=0x7f15b40ae120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:05.361 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.358+0000 7f15ba7e2640 10 monclient: _finish_hunting 0 2024-04-04T21:02:05.361 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.358+0000 7f15ba7e2640 1 monclient: found mon.noname-c 2024-04-04T21:02:05.361 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.358+0000 7f15ba7e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:05.361 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.358+0000 7f15ba7e2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:02:05.361 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.358+0000 7f15ba7e2640 1 -- 172.21.15.106:0/1114171746 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b40b0100 con 0x7f15b40af6d0 2024-04-04T21:02:05.361 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.360+0000 7f15ba7e2640 1 --2- 172.21.15.106:0/1114171746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15b40af6d0 0x7f15b40aed00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f15b413e120 tx=0x7f15ac02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5cd927ddf65ec29 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:05.365 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15b8fdf640 1 -- 172.21.15.106:0/1114171746 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac02ca80 con 0x7f15b40af6d0 2024-04-04T21:02:05.365 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15b8fdf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:05.365 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15b8fdf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:05.365 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15b8fdf640 10 monclient: dump: 2024-04-04T21:02:05.365 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:02:05.365 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:05.365 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:05.366 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:05.366 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:05.366 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:05.366 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:05.366 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:05.366 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:05.366 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:05.366 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15b8fdf640 10 monclient: _finish_auth 0 2024-04-04T21:02:05.366 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15b8fdf640 10 monclient: _check_auth_tickets 2024-04-04T21:02:05.366 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15b8fdf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:05.366 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15b8fdf640 1 -- 172.21.15.106:0/1114171746 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ac02cc20 con 0x7f15b40af6d0 2024-04-04T21:02:05.366 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15bc245640 5 monclient: authenticate success, global_id 4255 2024-04-04T21:02:05.366 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15bc245640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:05.366 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15b8fdf640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15bc245640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15bc245640 4 set_mon_vals no callback set 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15b8fdf640 1 -- 172.21.15.106:0/1114171746 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac0343f0 con 0x7f15b40af6d0 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15b8fdf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15b8fdf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.363+0000 7f15b8fdf640 10 monclient: dump: 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:05.367 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:05.368 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:05.368 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.364+0000 7f15bc245640 10 monclient: shutdown 2024-04-04T21:02:05.368 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.364+0000 7f15bc245640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:05.368 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.364+0000 7f15bc245640 1 -- 172.21.15.106:0/1114171746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15b40af6d0 msgr2=0x7f15b40aed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:05.368 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.364+0000 7f15bc245640 1 --2- 172.21.15.106:0/1114171746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15b40af6d0 0x7f15b40aed00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f15b413e120 tx=0x7f15ac02bee0 comp rx=0 tx=0).stop 2024-04-04T21:02:05.368 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.364+0000 7f15bc245640 1 -- 172.21.15.106:0/1114171746 shutdown_connections 2024-04-04T21:02:05.368 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.364+0000 7f15bc245640 1 --2- 172.21.15.106:0/1114171746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15b40af6d0 0x7f15b40aed00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:05.369 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.364+0000 7f15bc245640 1 --2- 172.21.15.106:0/1114171746 >> v2:172.21.15.106:3300/0 conn(0x7f15b406acc0 0x7f15b41403a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:05.369 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.364+0000 7f15bc245640 1 -- 172.21.15.106:0/1114171746 >> 172.21.15.106:0/1114171746 conn(0x7f15b40b4eb0 msgr2=0x7f15b40b52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:05.369 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.364+0000 7f15bc245640 1 -- 172.21.15.106:0/1114171746 shutdown_connections 2024-04-04T21:02:05.369 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.364+0000 7f15bc245640 1 -- 172.21.15.106:0/1114171746 wait complete. 2024-04-04T21:02:05.369 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.365+0000 7f15bc245640 10 monclient: build_initial_monmap 2024-04-04T21:02:05.369 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.365+0000 7f15bc245640 10 monclient: monmap: 2024-04-04T21:02:05.369 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:02:05.369 INFO:teuthology.misc.health.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:05.369 INFO:teuthology.misc.health.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:05.369 INFO:teuthology.misc.health.smithi106.stderr:created 0.000000 2024-04-04T21:02:05.369 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:05.369 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:05.370 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:05.370 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:05.370 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:05.371 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:05.371 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.365+0000 7f15bc245640 1 librados: starting msgr at 2024-04-04T21:02:05.371 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.365+0000 7f15bc245640 1 librados: starting objecter 2024-04-04T21:02:05.374 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.365+0000 7f15bc245640 1 Processor -- start 2024-04-04T21:02:05.374 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.365+0000 7f15bc245640 1 -- start start 2024-04-04T21:02:05.374 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.365+0000 7f15bc245640 1 librados: setting wanted keys 2024-04-04T21:02:05.374 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.365+0000 7f15bc245640 1 librados: calling monclient init 2024-04-04T21:02:05.374 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.365+0000 7f15bc245640 10 monclient: init 2024-04-04T21:02:05.374 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.365+0000 7f15bc245640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:05.374 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:02:05.374 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15b406acc0 0x7f15b41d56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:05.374 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 10 monclient(hunting): picked mon.noname-c con 0x7f15b406acc0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:05.374 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15b40af6d0 0x7f15b4142300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:05.375 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 10 monclient(hunting): picked mon.noname-b con 0x7f15b40af6d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:05.375 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15b41d60f0 0x7f15b4142840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:05.375 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 10 monclient(hunting): picked mon.noname-a con 0x7f15b41d60f0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:05.375 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:05.375 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b414aea0 con 0x7f15b41d60f0 2024-04-04T21:02:05.375 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:05.375 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b40b1610 con 0x7f15b40af6d0 2024-04-04T21:02:05.375 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:05.375 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b414b020 con 0x7f15b406acc0 2024-04-04T21:02:05.375 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:05.375 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15bc245640 10 monclient(hunting): authenticate will time out at 1374.633244s 2024-04-04T21:02:05.375 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15b97e0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15b40af6d0 0x7f15b4142300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:05.375 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15b9fe1640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15b406acc0 0x7f15b41d56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:05.375 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15b97e0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15b40af6d0 0x7f15b4142300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:46262/0 (socket says 172.21.15.106:46262) 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15b97e0640 1 -- 172.21.15.106:0/3876796785 learned_addr learned my addr 172.21.15.106:0/3876796785 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15b97e0640 10 monclient(hunting): get_auth_request con 0x7f15b40af6d0 auth_method 0 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15b97e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15b97e0640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.366+0000 7f15b97e0640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.367+0000 7f15b9fe1640 10 monclient(hunting): get_auth_request con 0x7f15b406acc0 auth_method 0 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.367+0000 7f15b9fe1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.367+0000 7f15b9fe1640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.367+0000 7f15b9fe1640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.367+0000 7f15b97e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.367+0000 7f15b97e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.367+0000 7f15b97e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.367+0000 7f15b9fe1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.367+0000 7f15b9fe1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:05.376 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.367+0000 7f15b9fe1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.368+0000 7f15ba7e2640 1 --2- 172.21.15.106:0/3876796785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15b41d60f0 0x7f15b4142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.369+0000 7f15ba7e2640 10 monclient(hunting): get_auth_request con 0x7f15b41d60f0 auth_method 0 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.369+0000 7f15ba7e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.369+0000 7f15ba7e2640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.369+0000 7f15ba7e2640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.370+0000 7f15b97e0640 10 monclient(hunting): handle_auth_done global_id 4258 payload 824 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.370+0000 7f15b97e0640 1 -- 172.21.15.106:0/3876796785 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15b406acc0 msgr2=0x7f15b41d56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.370+0000 7f15b97e0640 1 --2- 172.21.15.106:0/3876796785 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15b406acc0 0x7f15b41d56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.370+0000 7f15b97e0640 1 -- 172.21.15.106:0/3876796785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15b41d60f0 msgr2=0x7f15b4142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.370+0000 7f15b97e0640 1 --2- 172.21.15.106:0/3876796785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15b41d60f0 0x7f15b4142840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.370+0000 7f15b97e0640 10 monclient: _finish_hunting 0 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.370+0000 7f15b97e0640 1 monclient: found mon.noname-b 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.370+0000 7f15b97e0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.370+0000 7f15b97e0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:05.377 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.370+0000 7f15b97e0640 1 -- 172.21.15.106:0/3876796785 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b4143e10 con 0x7f15b40af6d0 2024-04-04T21:02:05.378 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.372+0000 7f15b97e0640 1 --2- 172.21.15.106:0/3876796785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15b40af6d0 0x7f15b4142300 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f15a400c3a0 tx=0x7f15a400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:05.378 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.372+0000 7f15aa7fc640 10 client.?.objecter ms_handle_connect 0x7f15b40af6d0 2024-04-04T21:02:05.378 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.372+0000 7f15aa7fc640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:05.379 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15aa7fc640 1 -- 172.21.15.106:0/3876796785 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a40052d0 con 0x7f15b40af6d0 2024-04-04T21:02:05.379 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:05.379 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:05.379 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15aa7fc640 10 monclient: dump: 2024-04-04T21:02:05.380 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:02:05.380 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:05.380 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:05.380 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:05.380 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:05.380 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:05.380 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:05.380 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:05.380 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:05.380 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:05.380 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15aa7fc640 10 monclient: _finish_auth 0 2024-04-04T21:02:05.380 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15aa7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:02:05.380 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:05.380 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15aa7fc640 1 -- 172.21.15.106:0/3876796785 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a4005490 con 0x7f15b40af6d0 2024-04-04T21:02:05.381 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15bc245640 5 monclient: authenticate success, global_id 4258 2024-04-04T21:02:05.381 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:05.381 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15bc245640 10 monclient: _renew_subs 2024-04-04T21:02:05.381 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15aa7fc640 1 -- 172.21.15.106:0/3876796785 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a4007260 con 0x7f15b40af6d0 2024-04-04T21:02:05.381 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15bc245640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:05.381 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15bc245640 1 -- 172.21.15.106:0/3876796785 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b4141de0 con 0x7f15b40af6d0 2024-04-04T21:02:05.381 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:05.381 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:05.381 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15aa7fc640 10 monclient: dump: 2024-04-04T21:02:05.381 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:02:05.381 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:05.381 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:05.381 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:05.381 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:05.382 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:05.382 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:05.382 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:05.382 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:05.382 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:05.382 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15abfff640 4 set_mon_vals no callback set 2024-04-04T21:02:05.382 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15bc245640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:05.382 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15bc245640 10 monclient: _renew_subs 2024-04-04T21:02:05.382 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15bc245640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:05.382 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15bc245640 1 -- 172.21.15.106:0/3876796785 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b41417b0 con 0x7f15b40af6d0 2024-04-04T21:02:05.382 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.378+0000 7f15bc245640 1 librados: init done 2024-04-04T21:02:05.383 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.379+0000 7f15bc245640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:05.383 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.379+0000 7f15bc245640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:05.383 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.379+0000 7f15bc245640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:05.383 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.379+0000 7f15bc245640 1 -- 172.21.15.106:0/3876796785 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b406a560 con 0x7f15b40af6d0 2024-04-04T21:02:05.397 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.397+0000 7f15aa7fc640 1 -- 172.21.15.106:0/3876796785 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f15a4007400 con 0x7f15b40af6d0 2024-04-04T21:02:05.398 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.398+0000 7f15aa7fc640 1 --2- 172.21.15.106:0/3876796785 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f159806ce50 0x7f159806f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:05.398 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.398+0000 7f15b9fe1640 1 --2- 172.21.15.106:0/3876796785 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f159806ce50 0x7f159806f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:05.398 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.398+0000 7f15b9fe1640 10 monclient: get_auth_request con 0x7f159806ce50 auth_method 0 2024-04-04T21:02:05.399 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.398+0000 7f15aa7fc640 10 client.4258.objecter ms_handle_connect 0x7f159806ce50 2024-04-04T21:02:05.399 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.398+0000 7f15b9fe1640 1 --2- 172.21.15.106:0/3876796785 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f159806ce50 0x7f159806f310 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f15b0009b40 tx=0x7f15b0009190 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:05.400 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.400+0000 7f15aa7fc640 1 -- 172.21.15.106:0/3876796785 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2743+0+0 (secure 0 0 0) 0x7f15a4056370 con 0x7f15b40af6d0 2024-04-04T21:02:05.400 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.400+0000 7f15aa7fc640 10 client.4258.objecter ms_dispatch 0x7f15b4000b90 osd_map(14..14 src has 1..14) v4 2024-04-04T21:02:05.400 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.400+0000 7f15aa7fc640 3 client.4258.objecter handle_osd_map got epochs [14,14] > 0 2024-04-04T21:02:05.400 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.400+0000 7f15aa7fc640 3 client.4258.objecter handle_osd_map decoding full epoch 14 2024-04-04T21:02:05.400 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.400+0000 7f15aa7fc640 20 client.4258.objecter dump_active .. 0 homeless 2024-04-04T21:02:05.653 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.653+0000 7f15aa7fc640 1 -- 172.21.15.106:0/3876796785 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f15a40582d0 con 0x7f15b40af6d0 2024-04-04T21:02:05.653 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.653+0000 7f15aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:05.653 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.653+0000 7f15aa7fc640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:05.880 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.880+0000 7f15bc245640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:05.880 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.880+0000 7f15bc245640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:05.880 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.880+0000 7f15bc245640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:05.881 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:05.880+0000 7f15bc245640 1 -- 172.21.15.106:0/3876796785 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15b4067ec0 con 0x7f15b40af6d0 2024-04-04T21:02:06.229 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.228+0000 7f15aa7fc640 1 -- 172.21.15.106:0/3876796785 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+10901 (secure 0 0 0) 0x7f15a405c370 con 0x7f15b40af6d0 2024-04-04T21:02:06.229 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.228+0000 7f15aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:06.229 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.228+0000 7f15aa7fc640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:06.229 INFO:teuthology.misc.health.smithi106.stdout: 2024-04-04T21:02:06.230 INFO:teuthology.misc.health.smithi106.stdout:{"epoch":15,"fsid":"8f89ce43-1bcb-41cf-9555-c40eeb258155","created":"2024-04-04T21:01:20.057083+0000","modified":"2024-04-04T21:02:05.463818+0000","last_up_change":"2024-04-04T21:02:05.463818+0000","last_in_change":"2024-04-04T21:01:25.642687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-04T21:02:04.746208+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"28337134-5253-454a-bcda-47267b05b2c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6819","nonce":2744620362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6821","nonce":2744620362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6825","nonce":2744620362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6823","nonce":2744620362}]},"public_addr":"172.21.15.106:6819/2744620362","cluster_addr":"172.21.15.106:6821/2744620362","heartbeat_back_addr":"172.21.15.106:6825/2744620362","heartbeat_front_addr":"172.21.15.106:6823/2744620362","state":["exists","up"]},{"osd":1,"uuid":"ae336a0e-2a90-4323-9f0f-1701fffe6b66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"773087e0-78c9-4d14-bf0a-fb605c21f92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07da9513-c746-46d1-995f-6f8e91338b84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c47d034c-1db6-4332-bf24-722fb2846808","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"588caa62-e773-44ae-94e5-ece54fe980d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6801","nonce":696438187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6803","nonce":696438187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6807","nonce":696438187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6805","nonce":696438187}]},"public_addr":"172.21.15.203:6801/696438187","cluster_addr":"172.21.15.203:6803/696438187","heartbeat_back_addr":"172.21.15.203:6807/696438187","heartbeat_front_addr":"172.21.15.203:6805/696438187","state":["exists","up"]},{"osd":6,"uuid":"ef776e12-7c80-4095-b173-5b2d24996f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6817","nonce":2758852830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6819","nonce":2758852830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6823","nonce":2758852830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6821","nonce":2758852830}]},"public_addr":"172.21.15.203:6817/2758852830","cluster_addr":"172.21.15.203:6819/2758852830","heartbeat_back_addr":"172.21.15.203:6823/2758852830","heartbeat_front_addr":"172.21.15.203:6821/2758852830","state":["exists","up"]},{"osd":7,"uuid":"53aafc58-6463-4aeb-8395-e14fcfcee5de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6809","nonce":4104640840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6811","nonce":4104640840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6815","nonce":4104640840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6813","nonce":4104640840}]},"public_addr":"172.21.15.203:6809/4104640840","cluster_addr":"172.21.15.203:6811/4104640840","heartbeat_back_addr":"172.21.15.203:6815/4104640840","heartbeat_front_addr":"172.21.15.203:6813/4104640840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:01.591956+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:01.851115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-04T21:02:06.232 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.231+0000 7f15bc245640 10 librados: watch_flush enter 2024-04-04T21:02:06.232 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.231+0000 7f15bc245640 10 librados: watch_flush exit 2024-04-04T21:02:06.232 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.231+0000 7f15bc245640 20 client.4258.objecter shutdown clearing up homeless session... 2024-04-04T21:02:06.232 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.231+0000 7f15bc245640 10 client.4258.objecter successfully canceled tick 2024-04-04T21:02:06.232 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.231+0000 7f15bc245640 1 -- 172.21.15.106:0/3876796785 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f159806ce50 msgr2=0x7f159806f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:06.232 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.231+0000 7f15bc245640 1 --2- 172.21.15.106:0/3876796785 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f159806ce50 0x7f159806f310 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f15b0009b40 tx=0x7f15b0009190 comp rx=0 tx=0).stop 2024-04-04T21:02:06.233 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.231+0000 7f15bc245640 10 monclient: shutdown 2024-04-04T21:02:06.233 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.231+0000 7f15bc245640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:06.233 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.231+0000 7f15bc245640 1 -- 172.21.15.106:0/3876796785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15b40af6d0 msgr2=0x7f15b4142300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:06.233 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.231+0000 7f15bc245640 1 --2- 172.21.15.106:0/3876796785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15b40af6d0 0x7f15b4142300 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f15a400c3a0 tx=0x7f15a400c860 comp rx=0 tx=0).stop 2024-04-04T21:02:06.233 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.232+0000 7f15bc245640 1 -- 172.21.15.106:0/3876796785 shutdown_connections 2024-04-04T21:02:06.233 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.232+0000 7f15bc245640 1 --2- 172.21.15.106:0/3876796785 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f159806ce50 0x7f159806f310 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:06.233 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.232+0000 7f15bc245640 1 --2- 172.21.15.106:0/3876796785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15b41d60f0 0x7f15b4142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:06.233 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.232+0000 7f15bc245640 1 --2- 172.21.15.106:0/3876796785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15b40af6d0 0x7f15b4142300 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:06.233 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.232+0000 7f15bc245640 1 --2- 172.21.15.106:0/3876796785 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15b406acc0 0x7f15b41d56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:06.233 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.232+0000 7f15bc245640 1 -- 172.21.15.106:0/3876796785 >> 172.21.15.106:0/3876796785 conn(0x7f15b40b4eb0 msgr2=0x7f15b40b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:06.233 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.232+0000 7f15bc245640 1 -- 172.21.15.106:0/3876796785 shutdown_connections 2024-04-04T21:02:06.233 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.232+0000 7f15bc245640 1 -- 172.21.15.106:0/3876796785 wait complete. 2024-04-04T21:02:06.233 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:06.232+0000 7f15bc245640 1 librados: shutdown 2024-04-04T21:02:06.247 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-04-04T21:02:10.729 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:02:10.435+0000 5dc3480 -1 bdev(0x18e77d10 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-04-04T21:02:10.933 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:02:10.835+0000 5dc3480 -1 bdev(0x18e77d10 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-04-04T21:02:11.003 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:02:10.880+0000 5dc3480 -1 bdev(0x18e77d10 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-04-04T21:02:11.074 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:02:10.933+0000 5dc3480 -1 bdev(0x18e77d10 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-04-04T21:02:11.074 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:02:10.944+0000 5dc3480 -1 bdev(0x18e77d10 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-04-04T21:02:11.107 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:02:11.032+0000 5dc3480 -1 bdev(0x18e77d10 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-04-04T21:02:11.190 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:02:11.075+0000 5dc3480 -1 bdev(0x18e77d10 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-04-04T21:02:11.191 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:02:11.108+0000 5dc3480 -1 bdev(0x18e77d10 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-04-04T21:02:12.374 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:02:12.371+0000 ac79640 -1 osd.4 0 waiting for initial osdmap 2024-04-04T21:02:12.839 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:02:12.824+0000 1425d640 -1 osd.4 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-04T21:02:13.054 DEBUG:teuthology.orchestra.run.smithi106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-04T21:02:13.153 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.152+0000 7fe800b06640 10 monclient: get_monmap_and_config 2024-04-04T21:02:13.154 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.152+0000 7fe800b06640 10 monclient: build_initial_monmap 2024-04-04T21:02:13.154 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.153+0000 7fe800b06640 10 monclient: monmap: 2024-04-04T21:02:13.154 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:02:13.154 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:13.154 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:02:13.154050+0000 2024-04-04T21:02:13.154 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:02:13.154050+0000 2024-04-04T21:02:13.154 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:13.154 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:13.154 INFO:teuthology.misc.health.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:13.154 INFO:teuthology.misc.health.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:13.155 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:13.155 INFO:teuthology.misc.health.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:13.155 INFO:teuthology.misc.health.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:13.155 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:13.155 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.153+0000 7fe800b06640 1 Processor -- start 2024-04-04T21:02:13.155 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.153+0000 7fe800b06640 1 -- start start 2024-04-04T21:02:13.155 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.153+0000 7fe800b06640 10 monclient: init 2024-04-04T21:02:13.157 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.154+0000 7fe800b06640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:13.158 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.154+0000 7fe800b06640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-04-04T21:02:13.158 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.154+0000 7fe800b06640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe7fc0b87e0 0x7fe7fc142650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:13.158 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.154+0000 7fe800b06640 10 monclient(hunting): picked mon.noname-c con 0x7fe7fc0b87e0 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:13.158 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.154+0000 7fe800b06640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe7fc0aa300 0x7fe7fc13f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:13.158 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.154+0000 7fe800b06640 10 monclient(hunting): picked mon.noname-a con 0x7fe7fc0aa300 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:13.158 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.154+0000 7fe800b06640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7fc142b90 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:13.158 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.154+0000 7fe800b06640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc0ab360 con 0x7fe7fc142b90 2024-04-04T21:02:13.158 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.154+0000 7fe800b06640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:13.158 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.154+0000 7fe800b06640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7fc0b8bc0 con 0x7fe7fc0aa300 2024-04-04T21:02:13.158 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.154+0000 7fe800b06640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:13.158 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.154+0000 7fe800b06640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7fc0ab640 con 0x7fe7fc0b87e0 2024-04-04T21:02:13.158 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.154+0000 7fe800b06640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:13.158 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.154+0000 7fe800b06640 10 monclient(hunting): authenticate will time out at 1382.421672s 2024-04-04T21:02:13.158 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.155+0000 7fe7f9d9b640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe7fc0aa300 0x7fe7fc13f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.155+0000 7fe7fa59c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe7fc0b87e0 0x7fe7fc142650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.155+0000 7fe7fa59c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe7fc0b87e0 0x7fe7fc142650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:46298/0 (socket says 172.21.15.106:46298) 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.155+0000 7fe7fa59c640 1 -- 172.21.15.106:0/4082224888 learned_addr learned my addr 172.21.15.106:0/4082224888 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.155+0000 7fe7fa59c640 10 monclient(hunting): get_auth_request con 0x7fe7fc0b87e0 auth_method 0 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.155+0000 7fe7fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.155+0000 7fe7fa59c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.155+0000 7fe7fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.155+0000 7fe7f9d9b640 10 monclient(hunting): get_auth_request con 0x7fe7fc0aa300 auth_method 0 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.155+0000 7fe7f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.155+0000 7fe7f9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.155+0000 7fe7f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.156+0000 7fe7fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.156+0000 7fe7fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.156+0000 7fe7fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:13.159 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.156+0000 7fe7f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:13.160 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.156+0000 7fe7f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:13.160 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.156+0000 7fe7f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:13.160 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.159+0000 7fe7fa59c640 10 monclient(hunting): handle_auth_done global_id 4270 payload 274 2024-04-04T21:02:13.161 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.159+0000 7fe7fa59c640 1 -- 172.21.15.106:0/4082224888 >> v2:172.21.15.106:3300/0 conn(0x7fe7fc0aa300 msgr2=0x7fe7fc13f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:13.161 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.159+0000 7fe7fa59c640 1 --2- 172.21.15.106:0/4082224888 >> v2:172.21.15.106:3300/0 conn(0x7fe7fc0aa300 0x7fe7fc13f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:13.161 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.159+0000 7fe7fa59c640 1 -- 172.21.15.106:0/4082224888 >> v1:172.21.15.203:6789/0 conn(0x7fe7fc142b90 legacy=0x7fe7fc145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:13.161 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.159+0000 7fe7fa59c640 10 monclient: _finish_hunting 0 2024-04-04T21:02:13.161 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.159+0000 7fe7fa59c640 1 monclient: found mon.noname-c 2024-04-04T21:02:13.161 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.159+0000 7fe7fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:13.161 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.159+0000 7fe7fa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:02:13.161 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.159+0000 7fe7fa59c640 1 -- 172.21.15.106:0/4082224888 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc140190 con 0x7fe7fc0b87e0 2024-04-04T21:02:13.161 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.159+0000 7fe7f9d9b640 1 --2- 172.21.15.106:0/4082224888 >> v2:172.21.15.106:3300/0 conn(0x7fe7fc0aa300 0x7fe7fc13f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-04T21:02:13.163 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.163+0000 7fe7fa59c640 1 --2- 172.21.15.106:0/4082224888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0b87e0 0x7fe7fc142650 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0b8540 tx=0x7fe7ec02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=186daade2e239a87 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:13.168 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe7f959a640 1 -- 172.21.15.106:0/4082224888 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec035020 con 0x7fe7fc0b87e0 2024-04-04T21:02:13.168 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe7f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:13.168 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe7f959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:13.168 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe7f959a640 10 monclient: dump: 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe7f959a640 10 monclient: _finish_auth 0 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe7f959a640 10 monclient: _check_auth_tickets 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe7f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe7f959a640 1 -- 172.21.15.106:0/4082224888 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec031870 con 0x7fe7fc0b87e0 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe7f959a640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:13.169 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe7f959a640 1 -- 172.21.15.106:0/4082224888 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec031c40 con 0x7fe7fc0b87e0 2024-04-04T21:02:13.170 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe800b06640 5 monclient: authenticate success, global_id 4270 2024-04-04T21:02:13.170 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe800b06640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:13.170 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe800b06640 4 set_mon_vals no callback set 2024-04-04T21:02:13.170 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe7f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:13.170 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe7f959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:13.170 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe7f959a640 10 monclient: dump: 2024-04-04T21:02:13.170 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:02:13.170 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:13.170 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:13.170 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:13.171 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:13.171 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:13.171 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:13.172 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:13.172 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:13.172 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:13.174 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe800b06640 10 monclient: shutdown 2024-04-04T21:02:13.174 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe800b06640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:13.174 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe800b06640 1 -- 172.21.15.106:0/4082224888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0b87e0 msgr2=0x7fe7fc142650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:13.175 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe800b06640 1 --2- 172.21.15.106:0/4082224888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0b87e0 0x7fe7fc142650 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0b8540 tx=0x7fe7ec02b910 comp rx=0 tx=0).stop 2024-04-04T21:02:13.175 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe800b06640 1 -- 172.21.15.106:0/4082224888 shutdown_connections 2024-04-04T21:02:13.175 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe800b06640 1 --2- 172.21.15.106:0/4082224888 >> v2:172.21.15.106:3300/0 conn(0x7fe7fc0aa300 0x7fe7fc13f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:13.175 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe800b06640 1 --2- 172.21.15.106:0/4082224888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0b87e0 0x7fe7fc142650 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:13.175 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.168+0000 7fe800b06640 1 -- 172.21.15.106:0/4082224888 >> 172.21.15.106:0/4082224888 conn(0x7fe7fc0727d0 msgr2=0x7fe7fc072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:13.175 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.169+0000 7fe800b06640 1 -- 172.21.15.106:0/4082224888 shutdown_connections 2024-04-04T21:02:13.175 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.169+0000 7fe800b06640 1 -- 172.21.15.106:0/4082224888 wait complete. 2024-04-04T21:02:13.175 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.169+0000 7fe800b06640 10 monclient: build_initial_monmap 2024-04-04T21:02:13.175 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.169+0000 7fe800b06640 10 monclient: monmap: 2024-04-04T21:02:13.175 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:02:13.175 INFO:teuthology.misc.health.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:13.175 INFO:teuthology.misc.health.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:13.175 INFO:teuthology.misc.health.smithi106.stderr:created 0.000000 2024-04-04T21:02:13.175 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.170+0000 7fe800b06640 1 librados: starting msgr at 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.170+0000 7fe800b06640 1 librados: starting objecter 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.170+0000 7fe800b06640 1 Processor -- start 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.170+0000 7fe800b06640 1 -- start start 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.170+0000 7fe800b06640 1 librados: setting wanted keys 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.170+0000 7fe800b06640 1 librados: calling monclient init 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.170+0000 7fe800b06640 10 monclient: init 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-04T21:02:13.176 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0aa300 0x7fe7fc1cf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 10 monclient(hunting): picked mon.noname-b con 0x7fe7fc0aa300 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7fc0b87e0 0x7fe7fc1cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 10 monclient(hunting): picked mon.noname-a con 0x7fe7fc0b87e0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7fc142b90 0x7fe7fc1ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 10 monclient(hunting): picked mon.noname-c con 0x7fe7fc142b90 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0a9b80 con 0x7fe7fc0b87e0 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0a9a00 con 0x7fe7fc0aa300 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc13faa0 con 0x7fe7fc142b90 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.171+0000 7fe800b06640 10 monclient(hunting): authenticate will time out at 1382.438802s 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fa59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0aa300 0x7fe7fc1cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:13.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fad9d640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7fc142b90 0x7fe7fc1ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fa59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0aa300 0x7fe7fc1cf210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:46312/0 (socket says 172.21.15.106:46312) 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fa59c640 1 -- 172.21.15.106:0/3704368188 learned_addr learned my addr 172.21.15.106:0/3704368188 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fa59c640 10 monclient(hunting): get_auth_request con 0x7fe7fc0aa300 auth_method 0 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fa59c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fad9d640 10 monclient(hunting): get_auth_request con 0x7fe7fc142b90 auth_method 0 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fad9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.172+0000 7fe7fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.173+0000 7fe7fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:13.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.173+0000 7fe7fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:13.179 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.173+0000 7fe7fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:13.180 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.175+0000 7fe7fa59c640 10 monclient(hunting): handle_auth_done global_id 4276 payload 824 2024-04-04T21:02:13.180 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.175+0000 7fe7fa59c640 1 -- 172.21.15.106:0/3704368188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7fc142b90 msgr2=0x7fe7fc1ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:13.180 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.175+0000 7fe7fa59c640 1 --2- 172.21.15.106:0/3704368188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7fc142b90 0x7fe7fc1ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:13.180 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.175+0000 7fe7fa59c640 1 -- 172.21.15.106:0/3704368188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7fc0b87e0 msgr2=0x7fe7fc1cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:13.180 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.175+0000 7fe7fa59c640 1 --2- 172.21.15.106:0/3704368188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7fc0b87e0 0x7fe7fc1cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:13.180 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.175+0000 7fe7fa59c640 10 monclient: _finish_hunting 0 2024-04-04T21:02:13.180 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.175+0000 7fe7fa59c640 1 monclient: found mon.noname-b 2024-04-04T21:02:13.180 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.175+0000 7fe7fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:13.180 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.175+0000 7fe7fa59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:13.180 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.175+0000 7fe7fa59c640 1 -- 172.21.15.106:0/3704368188 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc1ce530 con 0x7fe7fc0aa300 2024-04-04T21:02:13.180 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.178+0000 7fe7fa59c640 1 --2- 172.21.15.106:0/3704368188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0aa300 0x7fe7fc1cf210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec000c00 tx=0x7fe7ec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:13.180 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.178+0000 7fe7eaffd640 10 client.?.objecter ms_handle_connect 0x7fe7fc0aa300 2024-04-04T21:02:13.180 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.178+0000 7fe7eaffd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:13.184 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.181+0000 7fe7eaffd640 1 -- 172.21.15.106:0/3704368188 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec040070 con 0x7fe7fc0aa300 2024-04-04T21:02:13.184 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.181+0000 7fe7eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:13.184 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.181+0000 7fe7eaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:13.184 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.181+0000 7fe7eaffd640 10 monclient: dump: 2024-04-04T21:02:13.184 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:02:13.184 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:13.184 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:13.184 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:13.184 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:13.185 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:13.185 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:13.185 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:13.185 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:13.185 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:13.186 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.181+0000 7fe7eaffd640 10 monclient: _finish_auth 0 2024-04-04T21:02:13.186 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.181+0000 7fe7eaffd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:13.188 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.181+0000 7fe7eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:13.188 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.181+0000 7fe7eaffd640 1 -- 172.21.15.106:0/3704368188 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec030d00 con 0x7fe7fc0aa300 2024-04-04T21:02:13.188 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.181+0000 7fe7eaffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:13.188 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.181+0000 7fe7f8d99640 4 set_mon_vals no callback set 2024-04-04T21:02:13.188 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.182+0000 7fe7eaffd640 1 -- 172.21.15.106:0/3704368188 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec035040 con 0x7fe7fc0aa300 2024-04-04T21:02:13.188 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.182+0000 7fe800b06640 5 monclient: authenticate success, global_id 4276 2024-04-04T21:02:13.189 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.182+0000 7fe800b06640 10 monclient: _renew_subs 2024-04-04T21:02:13.189 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.182+0000 7fe800b06640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:13.189 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.182+0000 7fe800b06640 1 -- 172.21.15.106:0/3704368188 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc1ce7a0 con 0x7fe7fc0aa300 2024-04-04T21:02:13.189 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.182+0000 7fe7eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:13.189 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.182+0000 7fe7eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:13.189 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.182+0000 7fe7eaffd640 10 monclient: dump: 2024-04-04T21:02:13.190 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:02:13.190 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:13.190 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:13.190 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:13.190 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:13.190 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:13.190 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:13.190 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:13.190 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:13.191 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:13.191 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.182+0000 7fe800b06640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:13.191 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.182+0000 7fe800b06640 10 monclient: _renew_subs 2024-04-04T21:02:13.191 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.182+0000 7fe800b06640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:13.191 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.182+0000 7fe800b06640 1 -- 172.21.15.106:0/3704368188 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc0b10f0 con 0x7fe7fc0aa300 2024-04-04T21:02:13.191 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.182+0000 7fe800b06640 1 librados: init done 2024-04-04T21:02:13.191 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.183+0000 7fe800b06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:13.191 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.183+0000 7fe800b06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:13.191 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.183+0000 7fe800b06640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:13.191 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.183+0000 7fe800b06640 1 -- 172.21.15.106:0/3704368188 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc06dd50 con 0x7fe7fc0aa300 2024-04-04T21:02:13.201 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.201+0000 7fe7eaffd640 1 -- 172.21.15.106:0/3704368188 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe7ec04b060 con 0x7fe7fc0aa300 2024-04-04T21:02:13.202 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.201+0000 7fe7eaffd640 1 --2- 172.21.15.106:0/3704368188 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe7dc06ce60 0x7fe7dc06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:13.202 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.202+0000 7fe7f9d9b640 1 --2- 172.21.15.106:0/3704368188 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe7dc06ce60 0x7fe7dc06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:13.203 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.202+0000 7fe7f9d9b640 10 monclient: get_auth_request con 0x7fe7dc06ce60 auth_method 0 2024-04-04T21:02:13.203 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.202+0000 7fe7f9d9b640 1 --2- 172.21.15.106:0/3704368188 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe7dc06ce60 0x7fe7dc06f320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0006e80 tx=0x7fe7f0006dd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:13.203 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.202+0000 7fe7eaffd640 10 client.4276.objecter ms_handle_connect 0x7fe7dc06ce60 2024-04-04T21:02:13.207 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.207+0000 7fe7eaffd640 1 -- 172.21.15.106:0/3704368188 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3737+0+0 (secure 0 0 0) 0x7fe7ec078b70 con 0x7fe7fc0aa300 2024-04-04T21:02:13.207 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.207+0000 7fe7eaffd640 10 client.4276.objecter ms_dispatch 0x7fe7fc000b90 osd_map(17..17 src has 1..17) v4 2024-04-04T21:02:13.207 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.207+0000 7fe7eaffd640 3 client.4276.objecter handle_osd_map got epochs [17,17] > 0 2024-04-04T21:02:13.207 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.207+0000 7fe7eaffd640 3 client.4276.objecter handle_osd_map decoding full epoch 17 2024-04-04T21:02:13.207 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.207+0000 7fe7eaffd640 20 client.4276.objecter dump_active .. 0 homeless 2024-04-04T21:02:13.465 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.464+0000 7fe7eaffd640 1 -- 172.21.15.106:0/3704368188 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe7fc1ce7a0 con 0x7fe7fc0aa300 2024-04-04T21:02:13.465 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.464+0000 7fe7eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:13.465 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.464+0000 7fe7eaffd640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:13.622 INFO:tasks.ceph.osd.3.smithi106.stderr:2024-04-04T21:02:13.495+0000 5dc3480 -1 bdev(0x18aa2eb0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-04T21:02:13.697 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.696+0000 7fe800b06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:13.697 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.697+0000 7fe800b06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:13.697 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.697+0000 7fe800b06640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:13.697 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.697+0000 7fe800b06640 1 -- 172.21.15.106:0/3704368188 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7fc0689e0 con 0x7fe7fc0aa300 2024-04-04T21:02:13.736 INFO:tasks.ceph.osd.3.smithi106.stderr:2024-04-04T21:02:13.623+0000 5dc3480 -1 bdev(0x18aa2eb0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-04T21:02:13.776 INFO:tasks.ceph.osd.2.smithi106.stderr:2024-04-04T21:02:13.764+0000 5dc3480 -1 bdev(0x18e73300 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-04-04T21:02:13.796 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.795+0000 7fe7eaffd640 1 -- 172.21.15.106:0/3704368188 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+11429 (secure 0 0 0) 0x7fe7ec07b1d0 con 0x7fe7fc0aa300 2024-04-04T21:02:13.796 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.795+0000 7fe7eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:13.796 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.795+0000 7fe7eaffd640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:13.796 INFO:teuthology.misc.health.smithi106.stdout: 2024-04-04T21:02:13.797 INFO:teuthology.misc.health.smithi106.stdout:{"epoch":18,"fsid":"8f89ce43-1bcb-41cf-9555-c40eeb258155","created":"2024-04-04T21:01:20.057083+0000","modified":"2024-04-04T21:02:13.599416+0000","last_up_change":"2024-04-04T21:02:13.599416+0000","last_in_change":"2024-04-04T21:01:25.642687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-04T21:02:04.746208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"28337134-5253-454a-bcda-47267b05b2c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6819","nonce":2744620362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6821","nonce":2744620362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6825","nonce":2744620362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6823","nonce":2744620362}]},"public_addr":"172.21.15.106:6819/2744620362","cluster_addr":"172.21.15.106:6821/2744620362","heartbeat_back_addr":"172.21.15.106:6825/2744620362","heartbeat_front_addr":"172.21.15.106:6823/2744620362","state":["exists","up"]},{"osd":1,"uuid":"ae336a0e-2a90-4323-9f0f-1701fffe6b66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"773087e0-78c9-4d14-bf0a-fb605c21f92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07da9513-c746-46d1-995f-6f8e91338b84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c47d034c-1db6-4332-bf24-722fb2846808","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6825","nonce":2225997929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6827","nonce":2225997929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6831","nonce":2225997929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6829","nonce":2225997929}]},"public_addr":"172.21.15.203:6825/2225997929","cluster_addr":"172.21.15.203:6827/2225997929","heartbeat_back_addr":"172.21.15.203:6831/2225997929","heartbeat_front_addr":"172.21.15.203:6829/2225997929","state":["exists","up"]},{"osd":5,"uuid":"588caa62-e773-44ae-94e5-ece54fe980d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6801","nonce":696438187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6803","nonce":696438187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6807","nonce":696438187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6805","nonce":696438187}]},"public_addr":"172.21.15.203:6801/696438187","cluster_addr":"172.21.15.203:6803/696438187","heartbeat_back_addr":"172.21.15.203:6807/696438187","heartbeat_front_addr":"172.21.15.203:6805/696438187","state":["exists","up"]},{"osd":6,"uuid":"ef776e12-7c80-4095-b173-5b2d24996f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6817","nonce":2758852830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6819","nonce":2758852830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6823","nonce":2758852830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6821","nonce":2758852830}]},"public_addr":"172.21.15.203:6817/2758852830","cluster_addr":"172.21.15.203:6819/2758852830","heartbeat_back_addr":"172.21.15.203:6823/2758852830","heartbeat_front_addr":"172.21.15.203:6821/2758852830","state":["exists","up"]},{"osd":7,"uuid":"53aafc58-6463-4aeb-8395-e14fcfcee5de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6809","nonce":4104640840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6811","nonce":4104640840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6815","nonce":4104640840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6813","nonce":4104640840}]},"public_addr":"172.21.15.203:6809/4104640840","cluster_addr":"172.21.15.203:6811/4104640840","heartbeat_back_addr":"172.21.15.203:6815/4104640840","heartbeat_front_addr":"172.21.15.203:6813/4104640840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:03.982282+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:01.591956+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:02.315592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:01.851115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-04T21:02:13.798 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.797+0000 7fe800b06640 10 librados: watch_flush enter 2024-04-04T21:02:13.799 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.797+0000 7fe800b06640 10 librados: watch_flush exit 2024-04-04T21:02:13.799 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.797+0000 7fe800b06640 20 client.4276.objecter shutdown clearing up homeless session... 2024-04-04T21:02:13.799 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.797+0000 7fe800b06640 10 client.4276.objecter successfully canceled tick 2024-04-04T21:02:13.799 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.797+0000 7fe800b06640 1 -- 172.21.15.106:0/3704368188 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe7dc06ce60 msgr2=0x7fe7dc06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:13.799 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.797+0000 7fe800b06640 1 --2- 172.21.15.106:0/3704368188 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe7dc06ce60 0x7fe7dc06f320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0006e80 tx=0x7fe7f0006dd0 comp rx=0 tx=0).stop 2024-04-04T21:02:13.799 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.797+0000 7fe800b06640 10 monclient: shutdown 2024-04-04T21:02:13.799 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.797+0000 7fe800b06640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:13.799 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.797+0000 7fe800b06640 1 -- 172.21.15.106:0/3704368188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0aa300 msgr2=0x7fe7fc1cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:13.799 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.797+0000 7fe800b06640 1 --2- 172.21.15.106:0/3704368188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0aa300 0x7fe7fc1cf210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec000c00 tx=0x7fe7ec0046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:13.799 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.798+0000 7fe800b06640 1 -- 172.21.15.106:0/3704368188 shutdown_connections 2024-04-04T21:02:13.799 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.798+0000 7fe800b06640 1 --2- 172.21.15.106:0/3704368188 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe7dc06ce60 0x7fe7dc06f320 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:13.799 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.798+0000 7fe800b06640 1 --2- 172.21.15.106:0/3704368188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7fc142b90 0x7fe7fc1ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:13.799 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.798+0000 7fe800b06640 1 --2- 172.21.15.106:0/3704368188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7fc0b87e0 0x7fe7fc1cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:13.799 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.798+0000 7fe800b06640 1 --2- 172.21.15.106:0/3704368188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7fc0aa300 0x7fe7fc1cf210 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:13.800 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.798+0000 7fe800b06640 1 -- 172.21.15.106:0/3704368188 >> 172.21.15.106:0/3704368188 conn(0x7fe7fc0727d0 msgr2=0x7fe7fc13d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:13.800 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.798+0000 7fe800b06640 1 -- 172.21.15.106:0/3704368188 shutdown_connections 2024-04-04T21:02:13.800 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.798+0000 7fe800b06640 1 -- 172.21.15.106:0/3704368188 wait complete. 2024-04-04T21:02:13.800 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:13.798+0000 7fe800b06640 1 librados: shutdown 2024-04-04T21:02:13.810 DEBUG:teuthology.misc:5 of 8 OSDs are up 2024-04-04T21:02:13.833 INFO:tasks.ceph.osd.1.smithi106.stderr:2024-04-04T21:02:13.793+0000 5dc3480 -1 bdev(0x18b0c0a0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-04-04T21:02:13.836 INFO:tasks.ceph.osd.2.smithi106.stderr:2024-04-04T21:02:13.778+0000 5dc3480 -1 bdev(0x18e73300 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-04-04T21:02:13.838 INFO:tasks.ceph.osd.2.smithi106.stderr:2024-04-04T21:02:13.791+0000 5dc3480 -1 bdev(0x18e73300 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-04-04T21:02:13.867 INFO:tasks.ceph.osd.1.smithi106.stderr:2024-04-04T21:02:13.834+0000 5dc3480 -1 bdev(0x18b0c0a0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-04-04T21:02:13.960 INFO:tasks.ceph.osd.3.smithi106.stderr:2024-04-04T21:02:13.713+0000 5dc3480 -1 bdev(0x18aa2eb0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-04T21:02:14.036 INFO:tasks.ceph.osd.3.smithi106.stderr:2024-04-04T21:02:13.740+0000 5dc3480 -1 bdev(0x18aa2eb0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-04T21:02:14.037 INFO:tasks.ceph.osd.3.smithi106.stderr:2024-04-04T21:02:13.960+0000 5dc3480 -1 bdev(0x18aa2eb0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-04T21:02:14.051 INFO:tasks.ceph.osd.1.smithi106.stderr:2024-04-04T21:02:13.899+0000 5dc3480 -1 bdev(0x18b0c0a0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-04-04T21:02:14.051 INFO:tasks.ceph.osd.1.smithi106.stderr:2024-04-04T21:02:14.030+0000 5dc3480 -1 bdev(0x18b0c0a0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-04-04T21:02:14.118 INFO:tasks.ceph.osd.1.smithi106.stderr:2024-04-04T21:02:14.051+0000 5dc3480 -1 bdev(0x18b0c0a0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-04-04T21:02:14.277 INFO:tasks.ceph.osd.2.smithi106.stderr:2024-04-04T21:02:13.838+0000 5dc3480 -1 bdev(0x18e73300 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-04-04T21:02:14.318 INFO:tasks.ceph.osd.2.smithi106.stderr:2024-04-04T21:02:14.277+0000 5dc3480 -1 bdev(0x18e73300 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-04-04T21:02:14.416 INFO:tasks.ceph.osd.3.smithi106.stderr:2024-04-04T21:02:14.039+0000 5dc3480 -1 bdev(0x18aa2eb0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-04T21:02:14.470 INFO:tasks.ceph.osd.3.smithi106.stderr:2024-04-04T21:02:14.419+0000 5dc3480 -1 bdev(0x18aa2eb0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 2 2024-04-04T21:02:14.482 INFO:tasks.ceph.osd.3.smithi106.stderr:2024-04-04T21:02:14.470+0000 5dc3480 -1 bdev(0x18aa2eb0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 2 2024-04-04T21:02:14.592 INFO:tasks.ceph.osd.2.smithi106.stderr:2024-04-04T21:02:14.323+0000 5dc3480 -1 bdev(0x18e73300 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-04-04T21:02:14.623 INFO:tasks.ceph.osd.1.smithi106.stderr:2024-04-04T21:02:14.117+0000 5dc3480 -1 bdev(0x18b0c0a0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-04-04T21:02:15.448 INFO:tasks.ceph.osd.3.smithi106.stderr:2024-04-04T21:02:15.442+0000 ac79640 -1 osd.3 0 waiting for initial osdmap 2024-04-04T21:02:15.791 INFO:tasks.ceph.osd.1.smithi106.stderr:2024-04-04T21:02:15.784+0000 c48f640 -1 osd.1 0 waiting for initial osdmap 2024-04-04T21:02:15.799 INFO:tasks.ceph.osd.2.smithi106.stderr:2024-04-04T21:02:15.795+0000 ac79640 -1 osd.2 0 waiting for initial osdmap 2024-04-04T21:02:15.944 INFO:tasks.ceph.osd.3.smithi106.stderr:2024-04-04T21:02:15.930+0000 1425d640 -1 osd.3 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-04T21:02:16.273 INFO:tasks.ceph.osd.1.smithi106.stderr:2024-04-04T21:02:16.257+0000 1425d640 -1 osd.1 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-04T21:02:16.296 INFO:tasks.ceph.osd.2.smithi106.stderr:2024-04-04T21:02:16.282+0000 1425d640 -1 osd.2 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-04T21:02:17.773 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:01:09.290 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:17.833 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:01:09.348 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:17.848 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:09.330 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:17.849 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:01:09.365 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:17.899 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:09.381 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:17.909 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:01:09.425 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:18.132 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:09.614 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:18.142 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:01:09.659 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:18.169 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:01:09.686 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:20.617 DEBUG:teuthology.orchestra.run.smithi106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-04T21:02:20.709 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.708+0000 7f70aaeb7640 10 monclient: get_monmap_and_config 2024-04-04T21:02:20.709 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.708+0000 7f70aaeb7640 10 monclient: build_initial_monmap 2024-04-04T21:02:20.709 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.708+0000 7f70aaeb7640 10 monclient: monmap: 2024-04-04T21:02:20.709 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:02:20.709 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:20.709 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:02:20.709914+0000 2024-04-04T21:02:20.710 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:02:20.709914+0000 2024-04-04T21:02:20.710 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:20.710 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:20.710 INFO:teuthology.misc.health.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:20.710 INFO:teuthology.misc.health.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:20.710 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:20.710 INFO:teuthology.misc.health.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:20.710 INFO:teuthology.misc.health.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:20.710 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:20.711 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.709+0000 7f70aaeb7640 1 Processor -- start 2024-04-04T21:02:20.711 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.709+0000 7f70aaeb7640 1 -- start start 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.709+0000 7f70aaeb7640 10 monclient: init 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.710+0000 7f70aaeb7640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.710+0000 7f70aaeb7640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.710+0000 7f70aaeb7640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f70a40b51a0 0x7f70a40a9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.710+0000 7f70aaeb7640 10 monclient(hunting): picked mon.noname-a con 0x7f70a40b51a0 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.710+0000 7f70aaeb7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70a40b1d60 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.710+0000 7f70aaeb7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70a40aa2e0 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.710+0000 7f70aaeb7640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a4142cb0 con 0x7f70a40aa2e0 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.710+0000 7f70aaeb7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a40ab3d0 con 0x7f70a40b1d60 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.710+0000 7f70aaeb7640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.710+0000 7f70aaeb7640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a40b5580 con 0x7f70a40b51a0 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.710+0000 7f70aaeb7640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.710+0000 7f70aaeb7640 10 monclient(hunting): authenticate will time out at 1389.977501s 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.711+0000 7f70a3fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f70a40b1d60 0x7f70a41425a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:49542/0 (socket says 172.21.15.106:49542) 2024-04-04T21:02:20.712 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.711+0000 7f70a3fff640 1 -- 172.21.15.106:0/3035699112 learned_addr learned my addr 172.21.15.106:0/3035699112 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:20.715 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.715+0000 7f70a8c53640 1 --2- 172.21.15.106:0/3035699112 >> v2:172.21.15.106:3300/0 conn(0x7f70a40b51a0 0x7f70a40a9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:20.715 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.715+0000 7f70a8c53640 10 monclient(hunting): get_auth_request con 0x7f70a40b51a0 auth_method 0 2024-04-04T21:02:20.715 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.715+0000 7f70a8c53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:20.715 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.715+0000 7f70a8c53640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:20.715 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.715+0000 7f70a8c53640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:20.717 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.716+0000 7f70a8c53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:20.717 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.716+0000 7f70a8c53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:20.717 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.716+0000 7f70a8c53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:20.717 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.716+0000 7f70a37fe640 1 -- 172.21.15.106:0/3035699112 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845033357 0 0) 0x7f70a40ab3d0 con 0x7f70a40b1d60 2024-04-04T21:02:20.717 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.716+0000 7f70a37fe640 10 monclient(hunting): _negotiate 2024-04-04T21:02:20.717 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.716+0000 7f70a37fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:20.717 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.716+0000 7f70a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:20.717 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.716+0000 7f70a37fe640 10 monclient(hunting): my global_id is 4279 2024-04-04T21:02:20.717 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.716+0000 7f70a37fe640 1 -- 172.21.15.106:0/3035699112 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7094003520 con 0x7f70a40b1d60 2024-04-04T21:02:20.719 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.719+0000 7f70a8c53640 10 monclient(hunting): handle_auth_done global_id 4251 payload 274 2024-04-04T21:02:20.719 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.719+0000 7f70a8c53640 1 -- 172.21.15.106:0/3035699112 >> v1:172.21.15.203:6789/0 conn(0x7f70a40b1d60 legacy=0x7f70a41425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:20.719 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.719+0000 7f70a8c53640 1 -- 172.21.15.106:0/3035699112 >> v1:172.21.15.106:6789/0 conn(0x7f70a40aa2e0 legacy=0x7f70a413f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:20.719 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.719+0000 7f70a8c53640 10 monclient: _finish_hunting 0 2024-04-04T21:02:20.719 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.719+0000 7f70a8c53640 1 monclient: found mon.noname-a 2024-04-04T21:02:20.719 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.719+0000 7f70a8c53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:20.719 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.719+0000 7f70a8c53640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:20.720 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.719+0000 7f70a8c53640 1 -- 172.21.15.106:0/3035699112 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a413fe60 con 0x7f70a40b51a0 2024-04-04T21:02:20.722 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.722+0000 7f70a8c53640 1 --2- 172.21.15.106:0/3035699112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70a40b51a0 0x7f70a40a9d00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f70980097a0 tx=0x7f709802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f56f3c4ede2a485 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:20.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.727+0000 7f70a37fe640 1 -- 172.21.15.106:0/3035699112 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098035020 con 0x7f70a40b51a0 2024-04-04T21:02:20.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.727+0000 7f70a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:20.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.727+0000 7f70a37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:20.728 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.727+0000 7f70a37fe640 10 monclient: dump: 2024-04-04T21:02:20.728 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:02:20.728 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:20.728 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.727+0000 7f70a37fe640 10 monclient: _finish_auth 0 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.727+0000 7f70a37fe640 10 monclient: _check_auth_tickets 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.727+0000 7f70a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.727+0000 7f70a37fe640 1 -- 172.21.15.106:0/3035699112 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7098031830 con 0x7f70a40b51a0 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.727+0000 7f70a37fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.727+0000 7f70a37fe640 1 -- 172.21.15.106:0/3035699112 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098031c00 con 0x7f70a40b51a0 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.727+0000 7f70aaeb7640 5 monclient: authenticate success, global_id 4251 2024-04-04T21:02:20.729 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.727+0000 7f70aaeb7640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:20.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.727+0000 7f70aaeb7640 4 set_mon_vals no callback set 2024-04-04T21:02:20.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.728+0000 7f70a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:20.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.728+0000 7f70a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:20.730 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.728+0000 7f70a37fe640 10 monclient: dump: 2024-04-04T21:02:20.730 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:02:20.730 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:20.730 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:20.730 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:20.730 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:20.730 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:20.730 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:20.730 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:20.730 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:20.731 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:20.737 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.728+0000 7f70aaeb7640 10 monclient: shutdown 2024-04-04T21:02:20.737 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.728+0000 7f70aaeb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:20.737 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.728+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3035699112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70a40b51a0 msgr2=0x7f70a40a9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:20.737 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.728+0000 7f70aaeb7640 1 --2- 172.21.15.106:0/3035699112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70a40b51a0 0x7f70a40a9d00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f70980097a0 tx=0x7f709802b8d0 comp rx=0 tx=0).stop 2024-04-04T21:02:20.738 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.728+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3035699112 shutdown_connections 2024-04-04T21:02:20.738 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.728+0000 7f70aaeb7640 1 --2- 172.21.15.106:0/3035699112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70a40b51a0 0x7f70a40a9d00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:20.738 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.728+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3035699112 >> 172.21.15.106:0/3035699112 conn(0x7f70a40727a0 msgr2=0x7f70a4072bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:20.738 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.728+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3035699112 shutdown_connections 2024-04-04T21:02:20.738 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.728+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3035699112 wait complete. 2024-04-04T21:02:20.738 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.729+0000 7f70aaeb7640 10 monclient: build_initial_monmap 2024-04-04T21:02:20.738 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.729+0000 7f70aaeb7640 10 monclient: monmap: 2024-04-04T21:02:20.739 INFO:teuthology.misc.health.smithi106.stderr:epoch 0 2024-04-04T21:02:20.739 INFO:teuthology.misc.health.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:20.739 INFO:teuthology.misc.health.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:20.739 INFO:teuthology.misc.health.smithi106.stderr:created 0.000000 2024-04-04T21:02:20.739 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:20.739 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:20.739 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:20.739 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:20.739 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:20.739 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:20.739 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.729+0000 7f70aaeb7640 1 librados: starting msgr at 2024-04-04T21:02:20.739 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.729+0000 7f70aaeb7640 1 librados: starting objecter 2024-04-04T21:02:20.739 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.729+0000 7f70aaeb7640 1 Processor -- start 2024-04-04T21:02:20.739 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.729+0000 7f70aaeb7640 1 -- start start 2024-04-04T21:02:20.740 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.729+0000 7f70aaeb7640 1 librados: setting wanted keys 2024-04-04T21:02:20.740 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.729+0000 7f70aaeb7640 1 librados: calling monclient init 2024-04-04T21:02:20.740 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.729+0000 7f70aaeb7640 10 monclient: init 2024-04-04T21:02:20.740 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:20.740 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-04T21:02:20.740 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70a40aa2e0 0x7f70a41377e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:20.740 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 10 monclient(hunting): picked mon.noname-a con 0x7f70a40aa2e0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:20.740 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70a40b1d60 0x7f70a4137de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:20.740 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 10 monclient(hunting): picked mon.noname-b con 0x7f70a40b1d60 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:20.740 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70a40b51a0 0x7f70a4134fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:20.740 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 10 monclient(hunting): picked mon.noname-c con 0x7f70a40b51a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:20.740 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:20.740 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a40a9870 con 0x7f70a40aa2e0 2024-04-04T21:02:20.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:20.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a414e090 con 0x7f70a40b1d60 2024-04-04T21:02:20.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:20.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a414e210 con 0x7f70a40b51a0 2024-04-04T21:02:20.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:20.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.730+0000 7f70aaeb7640 10 monclient(hunting): authenticate will time out at 1389.997842s 2024-04-04T21:02:20.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a8c53640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70a40aa2e0 0x7f70a41377e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:20.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a3fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70a40b1d60 0x7f70a4137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:20.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a9454640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70a40b51a0 0x7f70a4134fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:20.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a8c53640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70a40aa2e0 0x7f70a41377e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55146/0 (socket says 172.21.15.106:55146) 2024-04-04T21:02:20.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a8c53640 1 -- 172.21.15.106:0/3409162448 learned_addr learned my addr 172.21.15.106:0/3409162448 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:20.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a8c53640 10 monclient(hunting): get_auth_request con 0x7f70a40aa2e0 auth_method 0 2024-04-04T21:02:20.741 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a8c53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:20.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a8c53640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:20.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a8c53640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:20.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a3fff640 10 monclient(hunting): get_auth_request con 0x7f70a40b1d60 auth_method 0 2024-04-04T21:02:20.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:20.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:20.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:20.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a9454640 10 monclient(hunting): get_auth_request con 0x7f70a40b51a0 auth_method 0 2024-04-04T21:02:20.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a9454640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:20.742 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a9454640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:20.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.731+0000 7f70a9454640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:20.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.732+0000 7f70a8c53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:20.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.732+0000 7f70a8c53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:20.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.732+0000 7f70a8c53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:20.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.732+0000 7f70a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:20.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.732+0000 7f70a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:20.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.732+0000 7f70a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:20.743 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.732+0000 7f70a9454640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:20.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.732+0000 7f70a9454640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:20.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.732+0000 7f70a9454640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:20.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.734+0000 7f70a3fff640 10 monclient(hunting): handle_auth_done global_id 4282 payload 824 2024-04-04T21:02:20.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.734+0000 7f70a3fff640 1 -- 172.21.15.106:0/3409162448 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70a40b51a0 msgr2=0x7f70a4134fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:20.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.734+0000 7f70a3fff640 1 --2- 172.21.15.106:0/3409162448 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70a40b51a0 0x7f70a4134fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:20.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.734+0000 7f70a3fff640 1 -- 172.21.15.106:0/3409162448 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70a40aa2e0 msgr2=0x7f70a41377e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:20.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.734+0000 7f70a3fff640 1 --2- 172.21.15.106:0/3409162448 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70a40aa2e0 0x7f70a41377e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:20.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.734+0000 7f70a3fff640 10 monclient: _finish_hunting 0 2024-04-04T21:02:20.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.734+0000 7f70a3fff640 1 monclient: found mon.noname-b 2024-04-04T21:02:20.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.734+0000 7f70a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:20.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.734+0000 7f70a3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:20.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.734+0000 7f70a3fff640 1 -- 172.21.15.106:0/3409162448 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a40ac230 con 0x7f70a40b1d60 2024-04-04T21:02:20.744 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.736+0000 7f70a3fff640 1 --2- 172.21.15.106:0/3409162448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70a40b1d60 0x7f70a4137de0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f709000c430 tx=0x7f709000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:20.745 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.736+0000 7f70a17fa640 10 client.?.objecter ms_handle_connect 0x7f70a40b1d60 2024-04-04T21:02:20.745 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.736+0000 7f70a17fa640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:20.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70a17fa640 1 -- 172.21.15.106:0/3409162448 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7090017070 con 0x7f70a40b1d60 2024-04-04T21:02:20.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:20.746 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70a17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:20.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70a17fa640 10 monclient: dump: 2024-04-04T21:02:20.747 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:02:20.747 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:20.747 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:20.747 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:20.747 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:20.747 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:20.747 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:20.747 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:20.747 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:20.747 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:20.747 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70a17fa640 10 monclient: _finish_auth 0 2024-04-04T21:02:20.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70a17fa640 10 monclient: _check_auth_tickets 2024-04-04T21:02:20.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:20.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70a17fa640 1 -- 172.21.15.106:0/3409162448 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7090005ce0 con 0x7f70a40b1d60 2024-04-04T21:02:20.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70aaeb7640 5 monclient: authenticate success, global_id 4282 2024-04-04T21:02:20.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70a17fa640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:20.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70a17fa640 1 -- 172.21.15.106:0/3409162448 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70900072f0 con 0x7f70a40b1d60 2024-04-04T21:02:20.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70aaeb7640 10 monclient: _renew_subs 2024-04-04T21:02:20.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70aaeb7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:20.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3409162448 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a40ac510 con 0x7f70a40b1d60 2024-04-04T21:02:20.748 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:20.749 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:20.749 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70a17fa640 10 monclient: dump: 2024-04-04T21:02:20.749 INFO:teuthology.misc.health.smithi106.stderr:epoch 1 2024-04-04T21:02:20.749 INFO:teuthology.misc.health.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:20.749 INFO:teuthology.misc.health.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:20.749 INFO:teuthology.misc.health.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:20.749 INFO:teuthology.misc.health.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:20.749 INFO:teuthology.misc.health.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:20.749 INFO:teuthology.misc.health.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:20.750 INFO:teuthology.misc.health.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:20.750 INFO:teuthology.misc.health.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:20.750 INFO:teuthology.misc.health.smithi106.stderr: 2024-04-04T21:02:20.750 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.740+0000 7f70a2ffd640 4 set_mon_vals no callback set 2024-04-04T21:02:20.750 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.741+0000 7f70aaeb7640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:20.750 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.741+0000 7f70aaeb7640 10 monclient: _renew_subs 2024-04-04T21:02:20.750 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.741+0000 7f70aaeb7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:20.750 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.741+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3409162448 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a4138a60 con 0x7f70a40b1d60 2024-04-04T21:02:20.751 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.741+0000 7f70aaeb7640 1 librados: init done 2024-04-04T21:02:20.751 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.741+0000 7f70aaeb7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:20.751 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.741+0000 7f70aaeb7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:20.751 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.741+0000 7f70aaeb7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:20.751 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.741+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3409162448 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a406a560 con 0x7f70a40b1d60 2024-04-04T21:02:20.993 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.992+0000 7f70a17fa640 1 -- 172.21.15.106:0/3409162448 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f709001a070 con 0x7f70a40b1d60 2024-04-04T21:02:20.993 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.992+0000 7f70a17fa640 1 --2- 172.21.15.106:0/3409162448 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f708806cee0 0x7f708806f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:20.993 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.992+0000 7f70a8c53640 1 --2- 172.21.15.106:0/3409162448 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f708806cee0 0x7f708806f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:20.993 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.993+0000 7f70a8c53640 10 monclient: get_auth_request con 0x7f708806cee0 auth_method 0 2024-04-04T21:02:20.993 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.993+0000 7f70a8c53640 1 --2- 172.21.15.106:0/3409162448 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f708806cee0 0x7f708806f3a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f709802b910 tx=0x7f709803c040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:20.994 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.993+0000 7f70a17fa640 10 client.4282.objecter ms_handle_connect 0x7f708806cee0 2024-04-04T21:02:20.994 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.994+0000 7f70a17fa640 1 -- 172.21.15.106:0/3409162448 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f709005e4f0 con 0x7f70a40b1d60 2024-04-04T21:02:20.994 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.994+0000 7f70a17fa640 10 client.4282.objecter ms_dispatch 0x7f70a4000b90 osd_map(22..22 src has 1..22) v4 2024-04-04T21:02:20.995 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.994+0000 7f70a17fa640 3 client.4282.objecter handle_osd_map got epochs [22,22] > 0 2024-04-04T21:02:20.995 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.994+0000 7f70a17fa640 3 client.4282.objecter handle_osd_map decoding full epoch 22 2024-04-04T21:02:20.995 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:20.994+0000 7f70a17fa640 20 client.4282.objecter dump_active .. 0 homeless 2024-04-04T21:02:21.005 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.005+0000 7f70a17fa640 1 -- 172.21.15.106:0/3409162448 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7090021020 con 0x7f70a40b1d60 2024-04-04T21:02:21.005 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.005+0000 7f70a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:21.005 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.005+0000 7f70a17fa640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:21.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.177+0000 7f70aaeb7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:21.177 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.177+0000 7f70aaeb7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:21.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.177+0000 7f70aaeb7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:21.178 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.177+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3409162448 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70a4138320 con 0x7f70a40b1d60 2024-04-04T21:02:21.202 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.202+0000 7f70a17fa640 1 -- 172.21.15.106:0/3409162448 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+12909 (secure 0 0 0) 0x7f7090007e20 con 0x7f70a40b1d60 2024-04-04T21:02:21.202 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.202+0000 7f70a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:21.202 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.202+0000 7f70a17fa640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:21.202 INFO:teuthology.misc.health.smithi106.stdout: 2024-04-04T21:02:21.202 INFO:teuthology.misc.health.smithi106.stdout:{"epoch":22,"fsid":"8f89ce43-1bcb-41cf-9555-c40eeb258155","created":"2024-04-04T21:01:20.057083+0000","modified":"2024-04-04T21:02:19.409702+0000","last_up_change":"2024-04-04T21:02:16.990959+0000","last_in_change":"2024-04-04T21:01:25.642687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-04T21:02:04.746208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"28337134-5253-454a-bcda-47267b05b2c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6819","nonce":2744620362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6821","nonce":2744620362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6825","nonce":2744620362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6823","nonce":2744620362}]},"public_addr":"172.21.15.106:6819/2744620362","cluster_addr":"172.21.15.106:6821/2744620362","heartbeat_back_addr":"172.21.15.106:6825/2744620362","heartbeat_front_addr":"172.21.15.106:6823/2744620362","state":["exists","up"]},{"osd":1,"uuid":"ae336a0e-2a90-4323-9f0f-1701fffe6b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6827","nonce":2321665741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6829","nonce":2321665741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6833","nonce":2321665741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6831","nonce":2321665741}]},"public_addr":"172.21.15.106:6827/2321665741","cluster_addr":"172.21.15.106:6829/2321665741","heartbeat_back_addr":"172.21.15.106:6833/2321665741","heartbeat_front_addr":"172.21.15.106:6831/2321665741","state":["exists","up"]},{"osd":2,"uuid":"773087e0-78c9-4d14-bf0a-fb605c21f92f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6811","nonce":459340769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6813","nonce":459340769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6817","nonce":459340769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6815","nonce":459340769}]},"public_addr":"172.21.15.106:6811/459340769","cluster_addr":"172.21.15.106:6813/459340769","heartbeat_back_addr":"172.21.15.106:6817/459340769","heartbeat_front_addr":"172.21.15.106:6815/459340769","state":["exists","up"]},{"osd":3,"uuid":"07da9513-c746-46d1-995f-6f8e91338b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6803","nonce":873500775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6805","nonce":873500775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6809","nonce":873500775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6807","nonce":873500775}]},"public_addr":"172.21.15.106:6803/873500775","cluster_addr":"172.21.15.106:6805/873500775","heartbeat_back_addr":"172.21.15.106:6809/873500775","heartbeat_front_addr":"172.21.15.106:6807/873500775","state":["exists","up"]},{"osd":4,"uuid":"c47d034c-1db6-4332-bf24-722fb2846808","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6825","nonce":2225997929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6827","nonce":2225997929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6831","nonce":2225997929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6829","nonce":2225997929}]},"public_addr":"172.21.15.203:6825/2225997929","cluster_addr":"172.21.15.203:6827/2225997929","heartbeat_back_addr":"172.21.15.203:6831/2225997929","heartbeat_front_addr":"172.21.15.203:6829/2225997929","state":["exists","up"]},{"osd":5,"uuid":"588caa62-e773-44ae-94e5-ece54fe980d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6801","nonce":696438187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6803","nonce":696438187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6807","nonce":696438187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6805","nonce":696438187}]},"public_addr":"172.21.15.203:6801/696438187","cluster_addr":"172.21.15.203:6803/696438187","heartbeat_back_addr":"172.21.15.203:6807/696438187","heartbeat_front_addr":"172.21.15.203:6805/696438187","state":["exists","up"]},{"osd":6,"uuid":"ef776e12-7c80-4095-b173-5b2d24996f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6817","nonce":2758852830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6819","nonce":2758852830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6823","nonce":2758852830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6821","nonce":2758852830}]},"public_addr":"172.21.15.203:6817/2758852830","cluster_addr":"172.21.15.203:6819/2758852830","heartbeat_back_addr":"172.21.15.203:6823/2758852830","heartbeat_front_addr":"172.21.15.203:6821/2758852830","state":["exists","up"]},{"osd":7,"uuid":"53aafc58-6463-4aeb-8395-e14fcfcee5de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6809","nonce":4104640840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6811","nonce":4104640840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6815","nonce":4104640840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6813","nonce":4104640840}]},"public_addr":"172.21.15.203:6809/4104640840","cluster_addr":"172.21.15.203:6811/4104640840","heartbeat_back_addr":"172.21.15.203:6815/4104640840","heartbeat_front_addr":"172.21.15.203:6813/4104640840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:03.982282+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:05.073358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:04.386488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:04.994186+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:02.522349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:01.591956+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:02.315592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:01.851115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-04T21:02:21.204 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 10 librados: watch_flush enter 2024-04-04T21:02:21.204 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 10 librados: watch_flush exit 2024-04-04T21:02:21.204 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 20 client.4282.objecter shutdown clearing up homeless session... 2024-04-04T21:02:21.204 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 10 client.4282.objecter successfully canceled tick 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3409162448 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f708806cee0 msgr2=0x7f708806f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 1 --2- 172.21.15.106:0/3409162448 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f708806cee0 0x7f708806f3a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f709802b910 tx=0x7f709803c040 comp rx=0 tx=0).stop 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 10 monclient: shutdown 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3409162448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70a40b1d60 msgr2=0x7f70a4137de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 1 --2- 172.21.15.106:0/3409162448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70a40b1d60 0x7f70a4137de0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f709000c430 tx=0x7f709000c8f0 comp rx=0 tx=0).stop 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3409162448 shutdown_connections 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 1 --2- 172.21.15.106:0/3409162448 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f708806cee0 0x7f708806f3a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 1 --2- 172.21.15.106:0/3409162448 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70a40b51a0 0x7f70a4134fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 1 --2- 172.21.15.106:0/3409162448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70a40b1d60 0x7f70a4137de0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 1 --2- 172.21.15.106:0/3409162448 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70a40aa2e0 0x7f70a41377e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3409162448 >> 172.21.15.106:0/3409162448 conn(0x7f70a40727a0 msgr2=0x7f70a4142ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3409162448 shutdown_connections 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 1 -- 172.21.15.106:0/3409162448 wait complete. 2024-04-04T21:02:21.205 INFO:teuthology.misc.health.smithi106.stderr:2024-04-04T21:02:21.203+0000 7f70aaeb7640 1 librados: shutdown 2024-04-04T21:02:21.213 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-04T21:02:21.214 INFO:tasks.ceph:Creating RBD pool 2024-04-04T21:02:21.214 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-04T21:02:21.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.314+0000 7f610ff0b640 10 monclient: get_monmap_and_config 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.315+0000 7f610ff0b640 10 monclient: build_initial_monmap 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.315+0000 7f610ff0b640 10 monclient: monmap: 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:21.316081+0000 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:21.316081+0000 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:21.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.315+0000 7f610ff0b640 1 Processor -- start 2024-04-04T21:02:21.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.315+0000 7f610ff0b640 1 -- start start 2024-04-04T21:02:21.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.315+0000 7f610ff0b640 10 monclient: init 2024-04-04T21:02:21.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.316+0000 7f610ff0b640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:21.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.316+0000 7f610ff0b640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-04-04T21:02:21.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.316+0000 7f610ff0b640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61080a9b60 0x7f61081430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:21.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.316+0000 7f610ff0b640 10 monclient(hunting): picked mon.noname-b con 0x7f61080a9b60 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:21.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.316+0000 7f610ff0b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f61081426f0 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:21.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.316+0000 7f610ff0b640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f610813f810 0x7f610813fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:21.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.316+0000 7f610ff0b640 10 monclient(hunting): picked mon.noname-a con 0x7f610813f810 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:21.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.316+0000 7f610ff0b640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61081435e0 con 0x7f61081426f0 2024-04-04T21:02:21.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.316+0000 7f610ff0b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:21.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.316+0000 7f610ff0b640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61080a9f40 con 0x7f610813f810 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.316+0000 7f610ff0b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.316+0000 7f610ff0b640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6108142c10 con 0x7f61080a9b60 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.316+0000 7f610ff0b640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.316+0000 7f610ff0b640 10 monclient(hunting): authenticate will time out at 1390.583670s 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.317+0000 7f610dca7640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61080a9b60 0x7f61081430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.317+0000 7f610e4a8640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f610813f810 0x7f610813fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.317+0000 7f610d4a6640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f61081426f0 0x7f61080b28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:54274/0 (socket says 172.21.15.106:54274) 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.317+0000 7f610d4a6640 1 -- 172.21.15.106:0/3551618129 learned_addr learned my addr 172.21.15.106:0/3551618129 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.317+0000 7f610dca7640 10 monclient(hunting): get_auth_request con 0x7f61080a9b60 auth_method 0 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.317+0000 7f610dca7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.317+0000 7f610dca7640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.317+0000 7f610dca7640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.318+0000 7f610dca7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.318+0000 7f610dca7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:21.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.318+0000 7f610dca7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:21.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.318+0000 7f610e4a8640 10 monclient(hunting): get_auth_request con 0x7f610813f810 auth_method 0 2024-04-04T21:02:21.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.318+0000 7f610e4a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:21.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.318+0000 7f610e4a8640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:21.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.318+0000 7f610e4a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:21.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.318+0000 7f610e4a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:21.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.318+0000 7f610e4a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:21.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.318+0000 7f610e4a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:21.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.319+0000 7f610dca7640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2024-04-04T21:02:21.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.319+0000 7f610dca7640 1 -- 172.21.15.106:0/3551618129 >> v2:172.21.15.106:3300/0 conn(0x7f610813f810 msgr2=0x7f610813fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:21.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.319+0000 7f610dca7640 1 --2- 172.21.15.106:0/3551618129 >> v2:172.21.15.106:3300/0 conn(0x7f610813f810 0x7f610813fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:21.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.319+0000 7f610dca7640 1 -- 172.21.15.106:0/3551618129 >> v1:172.21.15.106:6789/0 conn(0x7f61081426f0 legacy=0x7f61080b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:21.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.319+0000 7f610dca7640 10 monclient: _finish_hunting 0 2024-04-04T21:02:21.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.319+0000 7f610dca7640 1 monclient: found mon.noname-b 2024-04-04T21:02:21.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.319+0000 7f610dca7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:21.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.319+0000 7f610dca7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:21.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.319+0000 7f610dca7640 1 -- 172.21.15.106:0/3551618129 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61080b8c90 con 0x7f61080a9b60 2024-04-04T21:02:21.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.322+0000 7f610dca7640 1 --2- 172.21.15.106:0/3551618129 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61080a9b60 0x7f61081430a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f60f80097a0 tx=0x7f60f802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41795b7786e3e0cd server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:21.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.324+0000 7f610cca5640 1 -- 172.21.15.106:0/3551618129 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f80315a0 con 0x7f61080a9b60 2024-04-04T21:02:21.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.324+0000 7f610cca5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:21.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.324+0000 7f610cca5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.324+0000 7f610cca5640 10 monclient: dump: 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.324+0000 7f610cca5640 10 monclient: _finish_auth 0 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.324+0000 7f610cca5640 10 monclient: _check_auth_tickets 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.324+0000 7f610cca5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.324+0000 7f610ff0b640 5 monclient: authenticate success, global_id 4238 2024-04-04T21:02:21.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.324+0000 7f610ff0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:21.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.325+0000 7f610cca5640 1 -- 172.21.15.106:0/3551618129 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f8037070 con 0x7f61080a9b60 2024-04-04T21:02:21.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.325+0000 7f610cca5640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:21.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.325+0000 7f610cca5640 1 -- 172.21.15.106:0/3551618129 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f8031960 con 0x7f61080a9b60 2024-04-04T21:02:21.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.325+0000 7f610ff0b640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:21.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.325+0000 7f610ff0b640 4 set_mon_vals no callback set 2024-04-04T21:02:21.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.325+0000 7f610cca5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:21.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.325+0000 7f610cca5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:21.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.325+0000 7f610cca5640 10 monclient: dump: 2024-04-04T21:02:21.328 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:21.328 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:21.328 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:21.328 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:21.328 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:21.328 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:21.328 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:21.328 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.326+0000 7f610ff0b640 10 monclient: shutdown 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.326+0000 7f610ff0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.326+0000 7f610ff0b640 1 -- 172.21.15.106:0/3551618129 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61080a9b60 msgr2=0x7f61081430a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.326+0000 7f610ff0b640 1 --2- 172.21.15.106:0/3551618129 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61080a9b60 0x7f61081430a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f60f80097a0 tx=0x7f60f802b840 comp rx=0 tx=0).stop 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.326+0000 7f610ff0b640 1 -- 172.21.15.106:0/3551618129 shutdown_connections 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.326+0000 7f610ff0b640 1 --2- 172.21.15.106:0/3551618129 >> v2:172.21.15.106:3300/0 conn(0x7f610813f810 0x7f610813fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.326+0000 7f610ff0b640 1 --2- 172.21.15.106:0/3551618129 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61080a9b60 0x7f61081430a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.326+0000 7f610ff0b640 1 -- 172.21.15.106:0/3551618129 >> 172.21.15.106:0/3551618129 conn(0x7f61080b7ec0 msgr2=0x7f61080b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.326+0000 7f610ff0b640 1 -- 172.21.15.106:0/3551618129 shutdown_connections 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.326+0000 7f610ff0b640 1 -- 172.21.15.106:0/3551618129 wait complete. 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.327+0000 7f610ff0b640 10 monclient: build_initial_monmap 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.327+0000 7f610ff0b640 10 monclient: monmap: 2024-04-04T21:02:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:21.330 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:21.330 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:21.330 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:21.330 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:21.330 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:21.330 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:21.330 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:21.330 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:21.330 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:21.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.327+0000 7f610ff0b640 1 librados: starting msgr at 2024-04-04T21:02:21.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.327+0000 7f610ff0b640 1 librados: starting objecter 2024-04-04T21:02:21.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.327+0000 7f610ff0b640 1 Processor -- start 2024-04-04T21:02:21.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.327+0000 7f610ff0b640 1 -- start start 2024-04-04T21:02:21.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.327+0000 7f610ff0b640 1 librados: setting wanted keys 2024-04-04T21:02:21.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.327+0000 7f610ff0b640 1 librados: calling monclient init 2024-04-04T21:02:21.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.327+0000 7f610ff0b640 10 monclient: init 2024-04-04T21:02:21.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:21.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:02:21.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61080a9b60 0x7f6108139db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:21.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 10 monclient(hunting): picked mon.noname-c con 0x7f61080a9b60 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:21.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f610813f810 0x7f6108136940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:21.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 10 monclient(hunting): picked mon.noname-b con 0x7f610813f810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:21.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6108133fe0 0x7f6108134470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:21.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 10 monclient(hunting): picked mon.noname-a con 0x7f6108133fe0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:21.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:21.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61080b2530 con 0x7f6108133fe0 2024-04-04T21:02:21.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:21.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61080b23b0 con 0x7f610813f810 2024-04-04T21:02:21.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:21.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610814d000 con 0x7f61080a9b60 2024-04-04T21:02:21.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:21.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610ff0b640 10 monclient(hunting): authenticate will time out at 1390.595343s 2024-04-04T21:02:21.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610e4a8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6108133fe0 0x7f6108134470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:21.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610dca7640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61080a9b60 0x7f6108139db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610d4a6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f610813f810 0x7f6108136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610e4a8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6108133fe0 0x7f6108134470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55170/0 (socket says 172.21.15.106:55170) 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610e4a8640 1 -- 172.21.15.106:0/2883396776 learned_addr learned my addr 172.21.15.106:0/2883396776 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610e4a8640 10 monclient(hunting): get_auth_request con 0x7f6108133fe0 auth_method 0 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610e4a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610e4a8640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.328+0000 7f610e4a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610dca7640 10 monclient(hunting): get_auth_request con 0x7f61080a9b60 auth_method 0 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610dca7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610dca7640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610dca7640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610d4a6640 10 monclient(hunting): get_auth_request con 0x7f610813f810 auth_method 0 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610d4a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610d4a6640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610d4a6640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:21.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610e4a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:21.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610e4a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:21.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610e4a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:21.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610dca7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:21.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610dca7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:21.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610dca7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:21.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610d4a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:21.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610d4a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:21.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.329+0000 7f610d4a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:21.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.332+0000 7f610e4a8640 10 monclient(hunting): handle_auth_done global_id 4263 payload 824 2024-04-04T21:02:21.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.332+0000 7f610e4a8640 1 -- 172.21.15.106:0/2883396776 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61080a9b60 msgr2=0x7f6108139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:21.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.332+0000 7f610e4a8640 1 --2- 172.21.15.106:0/2883396776 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61080a9b60 0x7f6108139db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:21.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.332+0000 7f610e4a8640 1 -- 172.21.15.106:0/2883396776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f610813f810 msgr2=0x7f6108136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:21.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.332+0000 7f610e4a8640 1 --2- 172.21.15.106:0/2883396776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f610813f810 0x7f6108136940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:21.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.332+0000 7f610e4a8640 10 monclient: _finish_hunting 0 2024-04-04T21:02:21.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.332+0000 7f610e4a8640 1 monclient: found mon.noname-a 2024-04-04T21:02:21.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.332+0000 7f610e4a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:21.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.332+0000 7f610e4a8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:21.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.332+0000 7f610e4a8640 1 -- 172.21.15.106:0/2883396776 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61081349b0 con 0x7f6108133fe0 2024-04-04T21:02:21.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.334+0000 7f610e4a8640 1 --2- 172.21.15.106:0/2883396776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6108133fe0 0x7f6108134470 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f610000efc0 tx=0x7f610000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:21.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.334+0000 7f60fe7fc640 10 client.?.objecter ms_handle_connect 0x7f6108133fe0 2024-04-04T21:02:21.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.334+0000 7f60fe7fc640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:21.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f60fe7fc640 1 -- 172.21.15.106:0/2883396776 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6100018020 con 0x7f6108133fe0 2024-04-04T21:02:21.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f60fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:21.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f60fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:21.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f60fe7fc640 10 monclient: dump: 2024-04-04T21:02:21.339 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:21.339 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:21.339 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:21.339 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:21.339 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:21.340 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:21.340 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:21.340 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:21.340 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:21.340 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:21.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f60fe7fc640 10 monclient: _finish_auth 0 2024-04-04T21:02:21.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f60fe7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:02:21.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f60fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:21.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f60fe7fc640 1 -- 172.21.15.106:0/2883396776 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6100014480 con 0x7f6108133fe0 2024-04-04T21:02:21.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f610ff0b640 5 monclient: authenticate success, global_id 4263 2024-04-04T21:02:21.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f610ff0b640 10 monclient: _renew_subs 2024-04-04T21:02:21.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f610ff0b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:21.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f610ff0b640 1 -- 172.21.15.106:0/2883396776 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6108136e80 con 0x7f6108133fe0 2024-04-04T21:02:21.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f60fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f60fe7fc640 1 -- 172.21.15.106:0/2883396776 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6100014870 con 0x7f6108133fe0 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f60fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f60fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f60fe7fc640 10 monclient: dump: 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:21.341 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.337+0000 7f60fffff640 4 set_mon_vals no callback set 2024-04-04T21:02:21.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.338+0000 7f610ff0b640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:21.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.338+0000 7f610ff0b640 10 monclient: _renew_subs 2024-04-04T21:02:21.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.338+0000 7f610ff0b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:21.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.338+0000 7f610ff0b640 1 -- 172.21.15.106:0/2883396776 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6108136210 con 0x7f6108133fe0 2024-04-04T21:02:21.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.338+0000 7f610ff0b640 1 librados: init done 2024-04-04T21:02:21.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.339+0000 7f610ff0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:21.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.339+0000 7f610ff0b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:21.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.339+0000 7f610ff0b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:21.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.339+0000 7f610ff0b640 1 -- 172.21.15.106:0/2883396776 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f610804f050 con 0x7f6108133fe0 2024-04-04T21:02:21.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.356+0000 7f60fe7fc640 1 -- 172.21.15.106:0/2883396776 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6100005d10 con 0x7f6108133fe0 2024-04-04T21:02:21.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.356+0000 7f60fe7fc640 1 --2- 172.21.15.106:0/2883396776 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f60ec06cee0 0x7f60ec06f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:21.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.356+0000 7f610dca7640 1 --2- 172.21.15.106:0/2883396776 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f60ec06cee0 0x7f60ec06f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:21.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.356+0000 7f610dca7640 10 monclient: get_auth_request con 0x7f60ec06cee0 auth_method 0 2024-04-04T21:02:21.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.357+0000 7f60fe7fc640 10 client.4263.objecter ms_handle_connect 0x7f60ec06cee0 2024-04-04T21:02:21.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.357+0000 7f610dca7640 1 --2- 172.21.15.106:0/2883396776 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f60ec06cee0 0x7f60ec06f3a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f60f8036000 tx=0x7f60f803d040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:21.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.358+0000 7f60fe7fc640 1 -- 172.21.15.106:0/2883396776 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f610005ff00 con 0x7f6108133fe0 2024-04-04T21:02:21.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.358+0000 7f60fe7fc640 10 client.4263.objecter ms_dispatch 0x7f6108000b90 osd_map(22..22 src has 1..22) v4 2024-04-04T21:02:21.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.358+0000 7f60fe7fc640 3 client.4263.objecter handle_osd_map got epochs [22,22] > 0 2024-04-04T21:02:21.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.358+0000 7f60fe7fc640 3 client.4263.objecter handle_osd_map decoding full epoch 22 2024-04-04T21:02:21.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.358+0000 7f60fe7fc640 20 client.4263.objecter dump_active .. 0 homeless 2024-04-04T21:02:21.613 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.612+0000 7f60fe7fc640 1 -- 172.21.15.106:0/2883396776 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6100022030 con 0x7f6108133fe0 2024-04-04T21:02:21.613 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.612+0000 7f60fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:21.613 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.612+0000 7f60fe7fc640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:21.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.810+0000 7f610ff0b640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-04T21:02:21.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.810+0000 7f610ff0b640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-04T21:02:21.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.810+0000 7f610ff0b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:21.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:21.810+0000 7f610ff0b640 1 -- 172.21.15.106:0/2883396776 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f6108067d40 con 0x7f6108133fe0 2024-04-04T21:02:22.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.770+0000 7f60fe7fc640 1 -- 172.21.15.106:0/2883396776 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v23) v1 ==== 109+0+0 (secure 0 0 0) 0x7f6100010040 con 0x7f6108133fe0 2024-04-04T21:02:22.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.770+0000 7f60fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-04T21:02:22.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.770+0000 7f60fe7fc640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-04-04T21:02:22.771 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'rbd' created 2024-04-04T21:02:22.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.772+0000 7f610ff0b640 10 librados: watch_flush enter 2024-04-04T21:02:22.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.772+0000 7f610ff0b640 10 librados: watch_flush exit 2024-04-04T21:02:22.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.772+0000 7f610ff0b640 20 client.4263.objecter shutdown clearing up homeless session... 2024-04-04T21:02:22.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.772+0000 7f610ff0b640 10 client.4263.objecter successfully canceled tick 2024-04-04T21:02:22.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.772+0000 7f610ff0b640 1 -- 172.21.15.106:0/2883396776 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f60ec06cee0 msgr2=0x7f60ec06f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:22.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.772+0000 7f610ff0b640 1 --2- 172.21.15.106:0/2883396776 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f60ec06cee0 0x7f60ec06f3a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f60f8036000 tx=0x7f60f803d040 comp rx=0 tx=0).stop 2024-04-04T21:02:22.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.772+0000 7f610ff0b640 10 monclient: shutdown 2024-04-04T21:02:22.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.772+0000 7f610ff0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:22.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.772+0000 7f610ff0b640 1 -- 172.21.15.106:0/2883396776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6108133fe0 msgr2=0x7f6108134470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:22.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.772+0000 7f610ff0b640 1 --2- 172.21.15.106:0/2883396776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6108133fe0 0x7f6108134470 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f610000efc0 tx=0x7f610000c490 comp rx=0 tx=0).stop 2024-04-04T21:02:22.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.773+0000 7f610ff0b640 1 -- 172.21.15.106:0/2883396776 shutdown_connections 2024-04-04T21:02:22.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.773+0000 7f610ff0b640 1 --2- 172.21.15.106:0/2883396776 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f60ec06cee0 0x7f60ec06f3a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:22.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.773+0000 7f610ff0b640 1 --2- 172.21.15.106:0/2883396776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6108133fe0 0x7f6108134470 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:22.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.773+0000 7f610ff0b640 1 --2- 172.21.15.106:0/2883396776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f610813f810 0x7f6108136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:22.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.773+0000 7f610ff0b640 1 --2- 172.21.15.106:0/2883396776 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61080a9b60 0x7f6108139db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:22.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.773+0000 7f610ff0b640 1 -- 172.21.15.106:0/2883396776 >> 172.21.15.106:0/2883396776 conn(0x7f61080b7ec0 msgr2=0x7f61080b6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:22.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.773+0000 7f610ff0b640 1 -- 172.21.15.106:0/2883396776 shutdown_connections 2024-04-04T21:02:22.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.773+0000 7f610ff0b640 1 -- 172.21.15.106:0/2883396776 wait complete. 2024-04-04T21:02:22.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.773+0000 7f610ff0b640 1 librados: shutdown 2024-04-04T21:02:22.784 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-04T21:02:22.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.901+0000 7f5ce2647640 10 monclient: get_monmap_and_config 2024-04-04T21:02:22.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.901+0000 7f5ce2647640 10 monclient: build_initial_monmap 2024-04-04T21:02:22.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.901+0000 7f5ce2647640 10 monclient: monmap: 2024-04-04T21:02:22.902 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:22.902 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:22.902 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:22.902758+0000 2024-04-04T21:02:22.902 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:22.902758+0000 2024-04-04T21:02:22.902 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:22.902 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:22.902 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:22.903 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:22.903 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:22.903 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:22.903 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:22.903 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:22.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.902+0000 7f5ce2647640 1 Processor -- start 2024-04-04T21:02:22.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.902+0000 7f5ce2647640 1 -- start start 2024-04-04T21:02:22.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.902+0000 7f5ce2647640 10 monclient: init 2024-04-04T21:02:22.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:22.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-04-04T21:02:22.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5cdc0b5360 0x7f5cdc13f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:22.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 10 monclient(hunting): picked mon.noname-a con 0x7f5cdc0b5360 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:22.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cdc13faa0 0x7f5cdc145b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:22.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 10 monclient(hunting): picked mon.noname-b con 0x7f5cdc13faa0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:22.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5cdc146090 0x7f5cdc0b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:22.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 10 monclient(hunting): picked mon.noname-c con 0x7f5cdc146090 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:22.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:22.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cdc0b5740 con 0x7f5cdc0b5360 2024-04-04T21:02:22.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:22.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cdc0a9e70 con 0x7f5cdc146090 2024-04-04T21:02:22.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc13fee0 con 0x7f5cdc13faa0 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.903+0000 7f5ce2647640 10 monclient(hunting): authenticate will time out at 1392.170680s 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdb7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cdc13faa0 0x7f5cdc145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5ce0be4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5cdc146090 0x7f5cdc0b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdbfff640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5cdc0b5360 0x7f5cdc13f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdb7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cdc13faa0 0x7f5cdc145b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:35080/0 (socket says 172.21.15.106:35080) 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdb7fe640 1 -- 172.21.15.106:0/1127710744 learned_addr learned my addr 172.21.15.106:0/1127710744 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdb7fe640 10 monclient(hunting): get_auth_request con 0x7f5cdc13faa0 auth_method 0 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5ce0be4640 10 monclient(hunting): get_auth_request con 0x7f5cdc146090 auth_method 0 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5ce0be4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5ce0be4640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:22.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5ce0be4640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:22.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdbfff640 10 monclient(hunting): get_auth_request con 0x7f5cdc0b5360 auth_method 0 2024-04-04T21:02:22.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:22.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdbfff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:22.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:22.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:22.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:22.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5cdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:22.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5ce0be4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:22.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5ce0be4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:22.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.904+0000 7f5ce0be4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:22.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.905+0000 7f5cdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:22.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.905+0000 7f5cdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:22.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.905+0000 7f5cdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:22.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.906+0000 7f5cdb7fe640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2024-04-04T21:02:22.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.906+0000 7f5cdb7fe640 1 -- 172.21.15.106:0/1127710744 >> v2:172.21.15.203:3300/0 conn(0x7f5cdc146090 msgr2=0x7f5cdc0b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:22.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.906+0000 7f5cdb7fe640 1 --2- 172.21.15.106:0/1127710744 >> v2:172.21.15.203:3300/0 conn(0x7f5cdc146090 0x7f5cdc0b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:22.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.906+0000 7f5cdb7fe640 1 -- 172.21.15.106:0/1127710744 >> v2:172.21.15.106:3300/0 conn(0x7f5cdc0b5360 msgr2=0x7f5cdc13f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:22.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.906+0000 7f5cdb7fe640 1 --2- 172.21.15.106:0/1127710744 >> v2:172.21.15.106:3300/0 conn(0x7f5cdc0b5360 0x7f5cdc13f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:22.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.906+0000 7f5cdb7fe640 10 monclient: _finish_hunting 0 2024-04-04T21:02:22.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.906+0000 7f5cdb7fe640 1 monclient: found mon.noname-b 2024-04-04T21:02:22.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.906+0000 7f5cdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:22.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.906+0000 7f5cdb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:22.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.906+0000 7f5cdb7fe640 1 -- 172.21.15.106:0/1127710744 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc0ab3e0 con 0x7f5cdc13faa0 2024-04-04T21:02:22.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.906+0000 7f5ce0be4640 1 --2- 172.21.15.106:0/1127710744 >> v2:172.21.15.203:3300/0 conn(0x7f5cdc146090 0x7f5cdc0b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-04T21:02:22.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.908+0000 7f5cdb7fe640 1 --2- 172.21.15.106:0/1127710744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cdc13faa0 0x7f5cdc145b50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc0b3d70 tx=0x7f5ccc02b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25842aac7a0b4525 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:22.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.911+0000 7f5cdaffd640 1 -- 172.21.15.106:0/1127710744 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc038070 con 0x7f5cdc13faa0 2024-04-04T21:02:22.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.911+0000 7f5cdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:22.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.911+0000 7f5cdaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:22.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.911+0000 7f5cdaffd640 10 monclient: dump: 2024-04-04T21:02:22.913 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:22.913 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:22.913 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:22.913 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:22.913 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:22.913 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:22.913 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:22.914 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:22.914 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:22.914 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:22.914 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.911+0000 7f5cdaffd640 10 monclient: _finish_auth 0 2024-04-04T21:02:22.914 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.911+0000 7f5cdaffd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:22.914 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.911+0000 7f5cdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:22.914 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.911+0000 7f5cdaffd640 1 -- 172.21.15.106:0/1127710744 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc02f050 con 0x7f5cdc13faa0 2024-04-04T21:02:22.914 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.911+0000 7f5cdaffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:22.914 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.911+0000 7f5cdaffd640 1 -- 172.21.15.106:0/1127710744 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc031970 con 0x7f5cdc13faa0 2024-04-04T21:02:22.914 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.911+0000 7f5cdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:22.914 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.911+0000 7f5cdaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:22.914 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.911+0000 7f5cdaffd640 10 monclient: dump: 2024-04-04T21:02:22.914 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:22.914 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.912+0000 7f5ce2647640 5 monclient: authenticate success, global_id 4244 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.912+0000 7f5ce2647640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.912+0000 7f5ce2647640 4 set_mon_vals no callback set 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.912+0000 7f5ce2647640 10 monclient: shutdown 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.912+0000 7f5ce2647640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.912+0000 7f5ce2647640 1 -- 172.21.15.106:0/1127710744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cdc13faa0 msgr2=0x7f5cdc145b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.912+0000 7f5ce2647640 1 --2- 172.21.15.106:0/1127710744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cdc13faa0 0x7f5cdc145b50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc0b3d70 tx=0x7f5ccc02b7f0 comp rx=0 tx=0).stop 2024-04-04T21:02:22.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.912+0000 7f5ce2647640 1 -- 172.21.15.106:0/1127710744 shutdown_connections 2024-04-04T21:02:22.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.912+0000 7f5ce2647640 1 --2- 172.21.15.106:0/1127710744 >> v2:172.21.15.203:3300/0 conn(0x7f5cdc146090 0x7f5cdc0b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:22.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.912+0000 7f5ce2647640 1 --2- 172.21.15.106:0/1127710744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cdc13faa0 0x7f5cdc145b50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:22.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.912+0000 7f5ce2647640 1 --2- 172.21.15.106:0/1127710744 >> v2:172.21.15.106:3300/0 conn(0x7f5cdc0b5360 0x7f5cdc13f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:22.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.912+0000 7f5ce2647640 1 -- 172.21.15.106:0/1127710744 >> 172.21.15.106:0/1127710744 conn(0x7f5cdc066e70 msgr2=0x7f5cdc067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:22.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.912+0000 7f5ce2647640 1 -- 172.21.15.106:0/1127710744 shutdown_connections 2024-04-04T21:02:22.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.912+0000 7f5ce2647640 1 -- 172.21.15.106:0/1127710744 wait complete. 2024-04-04T21:02:22.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.913+0000 7f5ce2647640 10 monclient: build_initial_monmap 2024-04-04T21:02:22.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.913+0000 7f5ce2647640 10 monclient: monmap: 2024-04-04T21:02:22.919 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:22.919 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:22.919 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:22.919 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:22.919 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:22.919 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:22.919 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:22.919 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:22.919 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:22.919 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:22.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.914+0000 7f5ce2647640 1 librados: starting msgr at 2024-04-04T21:02:22.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.914+0000 7f5ce2647640 1 librados: starting objecter 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.914+0000 7f5ce2647640 1 Processor -- start 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.914+0000 7f5ce2647640 1 -- start start 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.914+0000 7f5ce2647640 1 librados: setting wanted keys 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.914+0000 7f5ce2647640 1 librados: calling monclient init 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.914+0000 7f5ce2647640 10 monclient: init 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cdc0b5360 0x7f5cdc1cf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 10 monclient(hunting): picked mon.noname-a con 0x7f5cdc0b5360 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cdc13faa0 0x7f5cdc1cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 10 monclient(hunting): picked mon.noname-c con 0x7f5cdc13faa0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc146090 0x7f5cdc1ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 10 monclient(hunting): picked mon.noname-b con 0x7f5cdc146090 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc146520 con 0x7f5cdc0b5360 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc0aa300 con 0x7f5cdc146090 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc140110 con 0x7f5cdc13faa0 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5ce2647640 10 monclient(hunting): authenticate will time out at 1392.182726s 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5cdbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cdc0b5360 0x7f5cdc1cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.915+0000 7f5cdb7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cdc13faa0 0x7f5cdc1cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5ce0be4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc146090 0x7f5cdc1ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cdc0b5360 0x7f5cdc1cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55190/0 (socket says 172.21.15.106:55190) 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdbfff640 1 -- 172.21.15.106:0/1040340134 learned_addr learned my addr 172.21.15.106:0/1040340134 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdbfff640 10 monclient(hunting): get_auth_request con 0x7f5cdc0b5360 auth_method 0 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdbfff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:22.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdb7fe640 10 monclient(hunting): get_auth_request con 0x7f5cdc13faa0 auth_method 0 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5ce0be4640 10 monclient(hunting): get_auth_request con 0x7f5cdc146090 auth_method 0 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5ce0be4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5ce0be4640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5ce0be4640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5ce0be4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5ce0be4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5ce0be4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:22.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:22.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:22.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.916+0000 7f5cdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:22.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.919+0000 7f5ce0be4640 10 monclient(hunting): handle_auth_done global_id 4291 payload 824 2024-04-04T21:02:22.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.919+0000 7f5ce0be4640 1 -- 172.21.15.106:0/1040340134 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cdc13faa0 msgr2=0x7f5cdc1cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:22.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.919+0000 7f5ce0be4640 1 --2- 172.21.15.106:0/1040340134 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cdc13faa0 0x7f5cdc1cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:22.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.919+0000 7f5ce0be4640 1 -- 172.21.15.106:0/1040340134 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cdc0b5360 msgr2=0x7f5cdc1cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:22.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.919+0000 7f5ce0be4640 1 --2- 172.21.15.106:0/1040340134 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cdc0b5360 0x7f5cdc1cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:22.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.919+0000 7f5ce0be4640 10 monclient: _finish_hunting 0 2024-04-04T21:02:22.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.919+0000 7f5ce0be4640 1 monclient: found mon.noname-b 2024-04-04T21:02:22.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.919+0000 7f5ce0be4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:22.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.919+0000 7f5ce0be4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:22.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.919+0000 7f5ce0be4640 1 -- 172.21.15.106:0/1040340134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc0acf90 con 0x7f5cdc146090 2024-04-04T21:02:22.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.921+0000 7f5ce0be4640 1 --2- 172.21.15.106:0/1040340134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc146090 0x7f5cdc1ccf00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400efc0 tx=0x7f5cd400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:22.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.921+0000 7f5cd8ff9640 10 client.?.objecter ms_handle_connect 0x7f5cdc146090 2024-04-04T21:02:22.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.921+0000 7f5cd8ff9640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:22.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5cd8ff9640 1 -- 172.21.15.106:0/1040340134 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4018030 con 0x7f5cdc146090 2024-04-04T21:02:22.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5cd8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:22.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5cd8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:22.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5cd8ff9640 10 monclient: dump: 2024-04-04T21:02:22.926 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:22.926 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:22.926 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:22.926 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:22.926 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:22.926 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:22.927 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:22.927 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:22.927 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:22.927 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:22.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5cd8ff9640 10 monclient: _finish_auth 0 2024-04-04T21:02:22.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5cd8ff9640 10 monclient: _check_auth_tickets 2024-04-04T21:02:22.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5cd8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:22.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5cd8ff9640 1 -- 172.21.15.106:0/1040340134 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd400cd30 con 0x7f5cdc146090 2024-04-04T21:02:22.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5ce2647640 5 monclient: authenticate success, global_id 4291 2024-04-04T21:02:22.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5cd8ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:22.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5cd8ff9640 1 -- 172.21.15.106:0/1040340134 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4003bb0 con 0x7f5cdc146090 2024-04-04T21:02:22.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5ce2647640 10 monclient: _renew_subs 2024-04-04T21:02:22.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5ce2647640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:22.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5ce2647640 1 -- 172.21.15.106:0/1040340134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cdc0ad270 con 0x7f5cdc146090 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5cda7fc640 4 set_mon_vals no callback set 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5cd8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5cd8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5cd8ff9640 10 monclient: dump: 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5ce2647640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:22.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5ce2647640 10 monclient: _renew_subs 2024-04-04T21:02:22.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5ce2647640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:22.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5ce2647640 1 -- 172.21.15.106:0/1040340134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cdc0ad830 con 0x7f5cdc146090 2024-04-04T21:02:22.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.925+0000 7f5ce2647640 1 librados: init done 2024-04-04T21:02:22.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.926+0000 7f5ce2647640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:22.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.926+0000 7f5ce2647640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:22.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.926+0000 7f5ce2647640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:22.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.926+0000 7f5ce2647640 1 -- 172.21.15.106:0/1040340134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cdc069c40 con 0x7f5cdc146090 2024-04-04T21:02:22.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.943+0000 7f5cd8ff9640 1 -- 172.21.15.106:0/1040340134 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5cd4005d10 con 0x7f5cdc146090 2024-04-04T21:02:22.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.944+0000 7f5cd8ff9640 1 --2- 172.21.15.106:0/1040340134 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5cc806cee0 0x7f5cc806f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:22.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.944+0000 7f5cdbfff640 1 --2- 172.21.15.106:0/1040340134 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5cc806cee0 0x7f5cc806f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:22.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.944+0000 7f5cdbfff640 10 monclient: get_auth_request con 0x7f5cc806cee0 auth_method 0 2024-04-04T21:02:22.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.944+0000 7f5cd8ff9640 10 client.4291.objecter ms_handle_connect 0x7f5cc806cee0 2024-04-04T21:02:22.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.944+0000 7f5cdbfff640 1 --2- 172.21.15.106:0/1040340134 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5cc806cee0 0x7f5cc806f3a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009c00 tx=0x7f5cd0009190 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:22.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.946+0000 7f5cd8ff9640 1 -- 172.21.15.106:0/1040340134 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f5cd40099d0 con 0x7f5cdc146090 2024-04-04T21:02:22.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.946+0000 7f5cd8ff9640 10 client.4291.objecter ms_dispatch 0x7f5cdc000b90 osd_map(23..23 src has 1..23) v4 2024-04-04T21:02:22.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.946+0000 7f5cd8ff9640 3 client.4291.objecter handle_osd_map got epochs [23,23] > 0 2024-04-04T21:02:22.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.946+0000 7f5cd8ff9640 3 client.4291.objecter handle_osd_map decoding full epoch 23 2024-04-04T21:02:22.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:22.946+0000 7f5cd8ff9640 20 client.4291.objecter dump_active .. 0 homeless 2024-04-04T21:02:23.193 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.193+0000 7f5cd8ff9640 1 -- 172.21.15.106:0/1040340134 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5cd4010040 con 0x7f5cdc146090 2024-04-04T21:02:23.193 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.193+0000 7f5cd8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:23.193 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.193+0000 7f5cd8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:23.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.437+0000 7f5ce2647640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-04T21:02:23.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.438+0000 7f5ce2647640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-04T21:02:23.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.438+0000 7f5ce2647640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:23.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.438+0000 7f5ce2647640 1 -- 172.21.15.106:0/1040340134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5cdc067f20 con 0x7f5cdc146090 2024-04-04T21:02:23.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.924+0000 7f5cd8ff9640 1 -- 172.21.15.106:0/1040340134 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v24) v1 ==== 173+0+0 (secure 0 0 0) 0x7f5cd4059800 con 0x7f5cdc146090 2024-04-04T21:02:23.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.924+0000 7f5cd8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-04T21:02:23.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.924+0000 7f5cd8ff9640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-04-04T21:02:23.925 INFO:teuthology.orchestra.run.smithi106.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-04T21:02:23.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.926+0000 7f5ce2647640 10 librados: watch_flush enter 2024-04-04T21:02:23.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.926+0000 7f5ce2647640 10 librados: watch_flush exit 2024-04-04T21:02:23.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.926+0000 7f5ce2647640 20 client.4291.objecter shutdown clearing up homeless session... 2024-04-04T21:02:23.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.926+0000 7f5ce2647640 10 client.4291.objecter successfully canceled tick 2024-04-04T21:02:23.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 1 -- 172.21.15.106:0/1040340134 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5cc806cee0 msgr2=0x7f5cc806f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:23.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 1 --2- 172.21.15.106:0/1040340134 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5cc806cee0 0x7f5cc806f3a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009c00 tx=0x7f5cd0009190 comp rx=0 tx=0).stop 2024-04-04T21:02:23.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 10 monclient: shutdown 2024-04-04T21:02:23.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:23.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 1 -- 172.21.15.106:0/1040340134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc146090 msgr2=0x7f5cdc1ccf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:23.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 1 --2- 172.21.15.106:0/1040340134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc146090 0x7f5cdc1ccf00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400efc0 tx=0x7f5cd400c490 comp rx=0 tx=0).stop 2024-04-04T21:02:23.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 1 -- 172.21.15.106:0/1040340134 shutdown_connections 2024-04-04T21:02:23.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 1 --2- 172.21.15.106:0/1040340134 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5cc806cee0 0x7f5cc806f3a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:23.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 1 --2- 172.21.15.106:0/1040340134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc146090 0x7f5cdc1ccf00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:23.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 1 --2- 172.21.15.106:0/1040340134 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cdc13faa0 0x7f5cdc1cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:23.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 1 --2- 172.21.15.106:0/1040340134 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cdc0b5360 0x7f5cdc1cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:23.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 1 -- 172.21.15.106:0/1040340134 >> 172.21.15.106:0/1040340134 conn(0x7f5cdc066e70 msgr2=0x7f5cdc144010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:23.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 1 -- 172.21.15.106:0/1040340134 shutdown_connections 2024-04-04T21:02:23.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 1 -- 172.21.15.106:0/1040340134 wait complete. 2024-04-04T21:02:23.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:23.927+0000 7f5ce2647640 1 librados: shutdown 2024-04-04T21:02:23.938 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-04T21:02:23.938 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-04T21:02:23.938 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-04T21:02:23.939 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-04T21:02:23.939 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-04T21:02:24.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.044+0000 7f8fb160f640 10 monclient: get_monmap_and_config 2024-04-04T21:02:24.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.044+0000 7f8fb160f640 10 monclient: build_initial_monmap 2024-04-04T21:02:24.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.044+0000 7f8fb160f640 10 monclient: monmap: 2024-04-04T21:02:24.045 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:24.045 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:24.045 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:24.045781+0000 2024-04-04T21:02:24.045 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:24.045781+0000 2024-04-04T21:02:24.046 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:24.046 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:24.046 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:24.046 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:24.046 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:24.046 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:24.046 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:24.046 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:24.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.045+0000 7f8fb160f640 1 Processor -- start 2024-04-04T21:02:24.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.045+0000 7f8fb160f640 1 -- start start 2024-04-04T21:02:24.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.045+0000 7f8fb160f640 10 monclient: init 2024-04-04T21:02:24.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fb160f640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:24.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fb160f640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-04-04T21:02:24.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fb160f640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fac0a9a50 0x7f8fac0a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:24.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fb160f640 10 monclient(hunting): picked mon.noname-b con 0x7f8fac0a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:24.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fb160f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8fac13f6a0 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:24.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fb160f640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8fac13fac0 0x7f8fac142f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:24.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fb160f640 10 monclient(hunting): picked mon.noname-a con 0x7f8fac13fac0 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:24.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fb160f640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fac0ab750 con 0x7f8fac13f6a0 2024-04-04T21:02:24.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fb160f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:24.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fb160f640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fac1434d0 con 0x7f8fac13fac0 2024-04-04T21:02:24.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fb160f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:24.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fb160f640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac13ff50 con 0x7f8fac0a9a50 2024-04-04T21:02:24.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fb160f640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:24.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fb160f640 10 monclient(hunting): authenticate will time out at 1393.313306s 2024-04-04T21:02:24.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8faaffd640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fac0a9a50 0x7f8fac0a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:24.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.046+0000 7f8fab7fe640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8fac13fac0 0x7f8fac142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:24.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.047+0000 7f8faaffd640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fac0a9a50 0x7f8fac0a9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:60884/0 (socket says 172.21.15.106:60884) 2024-04-04T21:02:24.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.047+0000 7f8faaffd640 1 -- 172.21.15.106:0/1078067158 learned_addr learned my addr 172.21.15.106:0/1078067158 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:24.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.047+0000 7f8faaffd640 10 monclient(hunting): get_auth_request con 0x7f8fac0a9a50 auth_method 0 2024-04-04T21:02:24.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.047+0000 7f8faaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:24.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.047+0000 7f8faaffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:24.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.047+0000 7f8faaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:24.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.047+0000 7f8fab7fe640 10 monclient(hunting): get_auth_request con 0x7f8fac13fac0 auth_method 0 2024-04-04T21:02:24.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.047+0000 7f8fab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:24.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.047+0000 7f8fab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:24.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.047+0000 7f8fab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:24.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.047+0000 7f8faaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:24.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.047+0000 7f8faaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:24.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.047+0000 7f8faaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:24.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.048+0000 7f8fab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:24.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.048+0000 7f8fab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:24.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.048+0000 7f8fab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:24.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.049+0000 7f8faaffd640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2024-04-04T21:02:24.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.049+0000 7f8faaffd640 1 -- 172.21.15.106:0/1078067158 >> v2:172.21.15.106:3300/0 conn(0x7f8fac13fac0 msgr2=0x7f8fac142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:24.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.049+0000 7f8faaffd640 1 --2- 172.21.15.106:0/1078067158 >> v2:172.21.15.106:3300/0 conn(0x7f8fac13fac0 0x7f8fac142f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:24.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.049+0000 7f8faaffd640 1 -- 172.21.15.106:0/1078067158 >> v1:172.21.15.106:6789/0 conn(0x7f8fac13f6a0 legacy=0x7f8fac142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:24.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.049+0000 7f8faaffd640 10 monclient: _finish_hunting 0 2024-04-04T21:02:24.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.049+0000 7f8faaffd640 1 monclient: found mon.noname-b 2024-04-04T21:02:24.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.049+0000 7f8faaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:24.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.049+0000 7f8faaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:24.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.049+0000 7f8faaffd640 1 -- 172.21.15.106:0/1078067158 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac140120 con 0x7f8fac0a9a50 2024-04-04T21:02:24.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.051+0000 7f8faaffd640 1 --2- 172.21.15.106:0/1078067158 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fac0a9a50 0x7f8fac0a9e30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009870 tx=0x7f8f9c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a10b3b9b4cc8e91 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:24.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8faa7fc640 1 -- 172.21.15.106:0/1078067158 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c035020 con 0x7f8fac0a9a50 2024-04-04T21:02:24.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8faa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:24.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8faa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:24.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8faa7fc640 10 monclient: dump: 2024-04-04T21:02:24.056 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:24.056 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:24.056 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:24.056 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:24.056 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:24.056 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:24.056 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:24.056 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:24.056 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:24.056 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:24.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8faa7fc640 10 monclient: _finish_auth 0 2024-04-04T21:02:24.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8faa7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8faa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8faa7fc640 1 -- 172.21.15.106:0/1078067158 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9c031640 con 0x7f8fac0a9a50 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8fb160f640 5 monclient: authenticate success, global_id 4250 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8fb160f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8faa7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8fb160f640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8fb160f640 4 set_mon_vals no callback set 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8faa7fc640 1 -- 172.21.15.106:0/1078067158 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c02d040 con 0x7f8fac0a9a50 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8faa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8faa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8faa7fc640 10 monclient: dump: 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:24.057 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:24.058 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:24.058 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:24.058 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:24.058 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:24.058 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:24.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8fb160f640 10 monclient: shutdown 2024-04-04T21:02:24.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8fb160f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:24.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8fb160f640 1 -- 172.21.15.106:0/1078067158 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fac0a9a50 msgr2=0x7f8fac0a9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:24.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.055+0000 7f8fb160f640 1 --2- 172.21.15.106:0/1078067158 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fac0a9a50 0x7f8fac0a9e30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009870 tx=0x7f8f9c02b8f0 comp rx=0 tx=0).stop 2024-04-04T21:02:24.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.056+0000 7f8fb160f640 1 -- 172.21.15.106:0/1078067158 shutdown_connections 2024-04-04T21:02:24.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.056+0000 7f8fb160f640 1 --2- 172.21.15.106:0/1078067158 >> v2:172.21.15.106:3300/0 conn(0x7f8fac13fac0 0x7f8fac142f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:24.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.056+0000 7f8fb160f640 1 --2- 172.21.15.106:0/1078067158 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fac0a9a50 0x7f8fac0a9e30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:24.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.056+0000 7f8fb160f640 1 -- 172.21.15.106:0/1078067158 >> 172.21.15.106:0/1078067158 conn(0x7f8fac0b8030 msgr2=0x7f8fac0b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:24.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.056+0000 7f8fb160f640 1 -- 172.21.15.106:0/1078067158 shutdown_connections 2024-04-04T21:02:24.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.056+0000 7f8fb160f640 1 -- 172.21.15.106:0/1078067158 wait complete. 2024-04-04T21:02:24.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.057+0000 7f8fb160f640 10 monclient: build_initial_monmap 2024-04-04T21:02:24.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.057+0000 7f8fb160f640 10 monclient: monmap: 2024-04-04T21:02:24.062 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:24.062 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:24.062 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.057+0000 7f8fb160f640 1 librados: starting msgr at 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.057+0000 7f8fb160f640 1 librados: starting objecter 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.057+0000 7f8fb160f640 1 Processor -- start 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.057+0000 7f8fb160f640 1 -- start start 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.057+0000 7f8fb160f640 1 librados: setting wanted keys 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.057+0000 7f8fb160f640 1 librados: calling monclient init 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.057+0000 7f8fb160f640 10 monclient: init 2024-04-04T21:02:24.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:02:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8fac0a9a50 0x7f8fac14a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 10 monclient(hunting): picked mon.noname-a con 0x7f8fac0a9a50 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fac13f6a0 0x7f8fac14ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 10 monclient(hunting): picked mon.noname-c con 0x7f8fac13f6a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fac13fac0 0x7f8fac14b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 10 monclient(hunting): picked mon.noname-b con 0x7f8fac13fac0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac14cef0 con 0x7f8fac0a9a50 2024-04-04T21:02:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac0aa370 con 0x7f8fac13fac0 2024-04-04T21:02:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac14d070 con 0x7f8fac13f6a0 2024-04-04T21:02:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:24.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.058+0000 7f8fb160f640 10 monclient(hunting): authenticate will time out at 1393.325948s 2024-04-04T21:02:24.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.059+0000 7f8faaffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8fac0a9a50 0x7f8fac14a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:24.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.059+0000 7f8fa3fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fac13f6a0 0x7f8fac14ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:24.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.059+0000 7f8faaffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8fac0a9a50 0x7f8fac14a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34714/0 (socket says 172.21.15.106:34714) 2024-04-04T21:02:24.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.059+0000 7f8fab7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fac13fac0 0x7f8fac14b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:24.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.059+0000 7f8faaffd640 1 -- 172.21.15.106:0/3969864822 learned_addr learned my addr 172.21.15.106:0/3969864822 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:24.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.059+0000 7f8faaffd640 10 monclient(hunting): get_auth_request con 0x7f8fac0a9a50 auth_method 0 2024-04-04T21:02:24.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.059+0000 7f8faaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:24.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.059+0000 7f8faaffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:24.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.059+0000 7f8faaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:24.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.059+0000 7f8fa3fff640 10 monclient(hunting): get_auth_request con 0x7f8fac13f6a0 auth_method 0 2024-04-04T21:02:24.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.059+0000 7f8fa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:24.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.059+0000 7f8fa3fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:24.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.059+0000 7f8fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:24.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.060+0000 7f8faaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:24.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.060+0000 7f8faaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:24.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.060+0000 7f8faaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:24.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.060+0000 7f8fab7fe640 10 monclient(hunting): get_auth_request con 0x7f8fac13fac0 auth_method 0 2024-04-04T21:02:24.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.060+0000 7f8fab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:24.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.060+0000 7f8fab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:24.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.060+0000 7f8fab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:24.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.060+0000 7f8fa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:24.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.060+0000 7f8fa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:24.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.060+0000 7f8fa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:24.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.060+0000 7f8fab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:24.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.060+0000 7f8fab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:24.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.060+0000 7f8fab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:24.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.063+0000 7f8faaffd640 10 monclient(hunting): handle_auth_done global_id 4278 payload 824 2024-04-04T21:02:24.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.063+0000 7f8faaffd640 1 -- 172.21.15.106:0/3969864822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fac13f6a0 msgr2=0x7f8fac14ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:24.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.063+0000 7f8faaffd640 1 --2- 172.21.15.106:0/3969864822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fac13f6a0 0x7f8fac14ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:24.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.063+0000 7f8faaffd640 1 -- 172.21.15.106:0/3969864822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fac13fac0 msgr2=0x7f8fac14b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:24.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.063+0000 7f8faaffd640 1 --2- 172.21.15.106:0/3969864822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fac13fac0 0x7f8fac14b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:24.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.063+0000 7f8faaffd640 10 monclient: _finish_hunting 0 2024-04-04T21:02:24.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.063+0000 7f8faaffd640 1 monclient: found mon.noname-a 2024-04-04T21:02:24.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.063+0000 7f8faaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:24.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.063+0000 7f8faaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:24.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.063+0000 7f8faaffd640 1 -- 172.21.15.106:0/3969864822 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac14c580 con 0x7f8fac0a9a50 2024-04-04T21:02:24.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.063+0000 7f8fa3fff640 1 --2- 172.21.15.106:0/3969864822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fac13f6a0 0x7f8fac14ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-04T21:02:24.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.065+0000 7f8faaffd640 1 --2- 172.21.15.106:0/3969864822 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8fac0a9a50 0x7f8fac14a400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c000c00 tx=0x7f8f9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:24.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.065+0000 7f8fa37fe640 10 client.?.objecter ms_handle_connect 0x7f8fac0a9a50 2024-04-04T21:02:24.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.065+0000 7f8fa37fe640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:24.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fa37fe640 1 -- 172.21.15.106:0/3969864822 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c040070 con 0x7f8fac0a9a50 2024-04-04T21:02:24.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fa37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:24.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fa37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:24.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fa37fe640 10 monclient: dump: 2024-04-04T21:02:24.071 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:24.071 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:24.071 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:24.071 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:24.071 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:24.071 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:24.071 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:24.071 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:24.071 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fa37fe640 10 monclient: _finish_auth 0 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fa37fe640 10 monclient: _check_auth_tickets 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fa37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fa37fe640 1 -- 172.21.15.106:0/3969864822 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9c0399f0 con 0x7f8fac0a9a50 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fb160f640 5 monclient: authenticate success, global_id 4278 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fb160f640 10 monclient: _renew_subs 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fb160f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fb160f640 1 -- 172.21.15.106:0/3969864822 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fac14c8c0 con 0x7f8fac0a9a50 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fa37fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fa37fe640 1 -- 172.21.15.106:0/3969864822 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c035040 con 0x7f8fac0a9a50 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fa37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fa37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fa37fe640 10 monclient: dump: 2024-04-04T21:02:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:24.073 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:24.073 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:24.073 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:24.073 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:24.073 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:24.073 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:24.073 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:24.073 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:24.073 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:24.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fa9ffb640 4 set_mon_vals no callback set 2024-04-04T21:02:24.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fb160f640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:24.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fb160f640 10 monclient: _renew_subs 2024-04-04T21:02:24.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fb160f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:24.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fb160f640 1 -- 172.21.15.106:0/3969864822 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fac0abc50 con 0x7f8fac0a9a50 2024-04-04T21:02:24.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.069+0000 7f8fb160f640 1 librados: init done 2024-04-04T21:02:24.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.070+0000 7f8fb160f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:24.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.070+0000 7f8fb160f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:24.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.070+0000 7f8fb160f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:24.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.070+0000 7f8fb160f640 1 -- 172.21.15.106:0/3969864822 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fac149320 con 0x7f8fac0a9a50 2024-04-04T21:02:24.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.096+0000 7f8fa37fe640 1 -- 172.21.15.106:0/3969864822 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8f9c04b060 con 0x7f8fac0a9a50 2024-04-04T21:02:24.097 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.096+0000 7f8fa37fe640 1 --2- 172.21.15.106:0/3969864822 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f8f8c06ce60 0x7f8f8c06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:24.097 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.096+0000 7f8fa3fff640 1 --2- 172.21.15.106:0/3969864822 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f8f8c06ce60 0x7f8f8c06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:24.097 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.096+0000 7f8fa3fff640 10 monclient: get_auth_request con 0x7f8f8c06ce60 auth_method 0 2024-04-04T21:02:24.097 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.097+0000 7f8fa37fe640 10 client.4278.objecter ms_handle_connect 0x7f8f8c06ce60 2024-04-04T21:02:24.097 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.097+0000 7f8fa3fff640 1 --2- 172.21.15.106:0/3969864822 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f8f8c06ce60 0x7f8f8c06f320 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0b7100 tx=0x7f8f94009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:24.101 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.101+0000 7f8fa37fe640 1 -- 172.21.15.106:0/3969864822 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8f9c0786c0 con 0x7f8fac0a9a50 2024-04-04T21:02:24.101 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.101+0000 7f8fa37fe640 10 client.4278.objecter ms_dispatch 0x7f8fac000b90 osd_map(24..24 src has 1..24) v4 2024-04-04T21:02:24.102 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.101+0000 7f8fa37fe640 3 client.4278.objecter handle_osd_map got epochs [24,24] > 0 2024-04-04T21:02:24.102 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.101+0000 7f8fa37fe640 3 client.4278.objecter handle_osd_map decoding full epoch 24 2024-04-04T21:02:24.102 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.101+0000 7f8fa37fe640 20 client.4278.objecter dump_active .. 0 homeless 2024-04-04T21:02:24.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.360+0000 7f8fa37fe640 1 -- 172.21.15.106:0/3969864822 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8fac14c8c0 con 0x7f8fac0a9a50 2024-04-04T21:02:24.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.360+0000 7f8fa37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:24.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.360+0000 7f8fa37fe640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:24.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.603+0000 7f8fb160f640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-04-04T21:02:24.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.603+0000 7f8fb160f640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-04T21:02:24.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.603+0000 7f8fb160f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:24.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.603+0000 7f8fb160f640 1 -- 172.21.15.106:0/3969864822 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f8fac04f050 con 0x7f8fac0a9a50 2024-04-04T21:02:24.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.743+0000 7f8fa37fe640 1 -- 172.21.15.106:0/3969864822 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171789 (secure 0 0 0) 0x7f8fac14c580 con 0x7f8fac0a9a50 2024-04-04T21:02:25.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.743+0000 7f8fa37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-04T21:02:25.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.743+0000 7f8fa37fe640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:25.379 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:02:25.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 10 librados: watch_flush enter 2024-04-04T21:02:25.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 10 librados: watch_flush exit 2024-04-04T21:02:25.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 20 client.4278.objecter shutdown clearing up homeless session... 2024-04-04T21:02:25.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 10 client.4278.objecter successfully canceled tick 2024-04-04T21:02:25.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 1 -- 172.21.15.106:0/3969864822 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f8f8c06ce60 msgr2=0x7f8f8c06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:25.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 1 --2- 172.21.15.106:0/3969864822 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f8f8c06ce60 0x7f8f8c06f320 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0b7100 tx=0x7f8f94009040 comp rx=0 tx=0).stop 2024-04-04T21:02:25.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 10 monclient: shutdown 2024-04-04T21:02:25.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:25.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 1 -- 172.21.15.106:0/3969864822 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8fac0a9a50 msgr2=0x7f8fac14a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:25.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 1 --2- 172.21.15.106:0/3969864822 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8fac0a9a50 0x7f8fac14a400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c000c00 tx=0x7f8f9c0046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:25.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 1 -- 172.21.15.106:0/3969864822 shutdown_connections 2024-04-04T21:02:25.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 1 --2- 172.21.15.106:0/3969864822 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f8f8c06ce60 0x7f8f8c06f320 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:25.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 1 --2- 172.21.15.106:0/3969864822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fac13fac0 0x7f8fac14b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:25.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 1 --2- 172.21.15.106:0/3969864822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fac13f6a0 0x7f8fac14ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:25.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 1 --2- 172.21.15.106:0/3969864822 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8fac0a9a50 0x7f8fac14a400 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:25.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 1 -- 172.21.15.106:0/3969864822 >> 172.21.15.106:0/3969864822 conn(0x7f8fac0b8030 msgr2=0x7f8fac13e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:25.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 1 -- 172.21.15.106:0/3969864822 shutdown_connections 2024-04-04T21:02:25.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 1 -- 172.21.15.106:0/3969864822 wait complete. 2024-04-04T21:02:25.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:24.746+0000 7f8fb160f640 1 librados: shutdown 2024-04-04T21:02:25.387 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":5,"flags":0,"active_gid":4117,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":2457463546},{"type":"v1","addr":"172.21.15.106:6801","nonce":2457463546}]},"active_addr":"172.21.15.106:6801/2457463546","active_change":"2024-04-04T21:01:25.885216+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4107,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":1513937129}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":117150773}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":3949202854}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":1931735909}]}]} 2024-04-04T21:02:25.395 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-04T21:02:25.395 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-04T21:02:25.395 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-04T21:02:25.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.499+0000 7fb0b5f39640 10 monclient: get_monmap_and_config 2024-04-04T21:02:25.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.499+0000 7fb0b5f39640 10 monclient: build_initial_monmap 2024-04-04T21:02:25.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.500+0000 7fb0b5f39640 10 monclient: monmap: 2024-04-04T21:02:25.500 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:25.500 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:25.500 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:25.501057+0000 2024-04-04T21:02:25.501 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:25.501057+0000 2024-04-04T21:02:25.501 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:25.501 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:25.501 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:25.501 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:25.501 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:25.501 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:25.501 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:25.501 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:25.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.500+0000 7fb0b5f39640 1 Processor -- start 2024-04-04T21:02:25.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.500+0000 7fb0b5f39640 1 -- start start 2024-04-04T21:02:25.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.500+0000 7fb0b5f39640 10 monclient: init 2024-04-04T21:02:25.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:25.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-04T21:02:25.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb0b00a9b60 0x7fb0b01430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:25.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 10 monclient(hunting): picked mon.noname-c con 0x7fb0b00a9b60 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:25.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb0b01426f0 0x7fb0b00b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:25.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 10 monclient(hunting): picked mon.noname-a con 0x7fb0b01426f0 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:25.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0b013f810 0x7fb0b013fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:25.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 10 monclient(hunting): picked mon.noname-b con 0x7fb0b013f810 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:25.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:25.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b00a9f40 con 0x7fb0b01426f0 2024-04-04T21:02:25.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:25.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b0142c10 con 0x7fb0b00a9b60 2024-04-04T21:02:25.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:25.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b01435e0 con 0x7fb0b013f810 2024-04-04T21:02:25.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:25.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0b5f39640 10 monclient(hunting): authenticate will time out at 1394.768555s 2024-04-04T21:02:25.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.501+0000 7fb0aeffd640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb0b01426f0 0x7fb0b00b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:25.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0affff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0b013f810 0x7fb0b013fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:25.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0af7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb0b00a9b60 0x7fb0b01430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:25.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0aeffd640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb0b01426f0 0x7fb0b00b28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34722/0 (socket says 172.21.15.106:34722) 2024-04-04T21:02:25.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0aeffd640 1 -- 172.21.15.106:0/2421351087 learned_addr learned my addr 172.21.15.106:0/2421351087 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:25.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0aeffd640 10 monclient(hunting): get_auth_request con 0x7fb0b01426f0 auth_method 0 2024-04-04T21:02:25.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:25.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0aeffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:25.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:25.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0affff640 10 monclient(hunting): get_auth_request con 0x7fb0b013f810 auth_method 0 2024-04-04T21:02:25.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:25.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0affff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:25.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:25.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0af7fe640 10 monclient(hunting): get_auth_request con 0x7fb0b00a9b60 auth_method 0 2024-04-04T21:02:25.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:25.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0af7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:25.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:25.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:25.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:25.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:25.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:25.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.502+0000 7fb0af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:25.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.503+0000 7fb0af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:25.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.503+0000 7fb0affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:25.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.503+0000 7fb0affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:25.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.503+0000 7fb0affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:25.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.504+0000 7fb0affff640 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2024-04-04T21:02:25.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.504+0000 7fb0affff640 1 -- 172.21.15.106:0/2421351087 >> v2:172.21.15.203:3300/0 conn(0x7fb0b00a9b60 msgr2=0x7fb0b01430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:25.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.504+0000 7fb0affff640 1 --2- 172.21.15.106:0/2421351087 >> v2:172.21.15.203:3300/0 conn(0x7fb0b00a9b60 0x7fb0b01430a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:25.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.504+0000 7fb0affff640 1 -- 172.21.15.106:0/2421351087 >> v2:172.21.15.106:3300/0 conn(0x7fb0b01426f0 msgr2=0x7fb0b00b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:25.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.504+0000 7fb0affff640 1 --2- 172.21.15.106:0/2421351087 >> v2:172.21.15.106:3300/0 conn(0x7fb0b01426f0 0x7fb0b00b28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:25.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.504+0000 7fb0affff640 10 monclient: _finish_hunting 0 2024-04-04T21:02:25.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.504+0000 7fb0affff640 1 monclient: found mon.noname-b 2024-04-04T21:02:25.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.504+0000 7fb0affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:25.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.504+0000 7fb0affff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:25.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.504+0000 7fb0affff640 1 -- 172.21.15.106:0/2421351087 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b00b2e00 con 0x7fb0b013f810 2024-04-04T21:02:25.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.504+0000 7fb0af7fe640 1 --2- 172.21.15.106:0/2421351087 >> v2:172.21.15.203:3300/0 conn(0x7fb0b00a9b60 0x7fb0b01430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-04T21:02:25.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.504+0000 7fb0aeffd640 1 --2- 172.21.15.106:0/2421351087 >> v2:172.21.15.106:3300/0 conn(0x7fb0b01426f0 0x7fb0b00b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-04T21:02:25.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.506+0000 7fb0affff640 1 --2- 172.21.15.106:0/2421351087 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0b013f810 0x7fb0b013fca0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00b59d0 tx=0x7fb0a802bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9da93c4b2cc87d97 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:25.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0ae7fc640 1 -- 172.21.15.106:0/2421351087 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8039070 con 0x7fb0b013f810 2024-04-04T21:02:25.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:25.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:25.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0ae7fc640 10 monclient: dump: 2024-04-04T21:02:25.521 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:25.521 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:25.521 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:25.521 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:25.521 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:25.521 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:25.522 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:25.522 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:25.522 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:25.522 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:25.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0ae7fc640 10 monclient: _finish_auth 0 2024-04-04T21:02:25.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0ae7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:02:25.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:25.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0ae7fc640 1 -- 172.21.15.106:0/2421351087 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a802cb40 con 0x7fb0b013f810 2024-04-04T21:02:25.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0b5f39640 5 monclient: authenticate success, global_id 4256 2024-04-04T21:02:25.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0b5f39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:25.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:25.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0b5f39640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:25.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0b5f39640 4 set_mon_vals no callback set 2024-04-04T21:02:25.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0ae7fc640 1 -- 172.21.15.106:0/2421351087 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8034440 con 0x7fb0b013f810 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.510+0000 7fb0ae7fc640 10 monclient: dump: 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.511+0000 7fb0b5f39640 10 monclient: shutdown 2024-04-04T21:02:25.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.511+0000 7fb0b5f39640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.511+0000 7fb0b5f39640 1 -- 172.21.15.106:0/2421351087 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0b013f810 msgr2=0x7fb0b013fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.511+0000 7fb0b5f39640 1 --2- 172.21.15.106:0/2421351087 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0b013f810 0x7fb0b013fca0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00b59d0 tx=0x7fb0a802bd90 comp rx=0 tx=0).stop 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.511+0000 7fb0b5f39640 1 -- 172.21.15.106:0/2421351087 shutdown_connections 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.511+0000 7fb0b5f39640 1 --2- 172.21.15.106:0/2421351087 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0b013f810 0x7fb0b013fca0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.511+0000 7fb0b5f39640 1 --2- 172.21.15.106:0/2421351087 >> v2:172.21.15.106:3300/0 conn(0x7fb0b01426f0 0x7fb0b00b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.511+0000 7fb0b5f39640 1 --2- 172.21.15.106:0/2421351087 >> v2:172.21.15.203:3300/0 conn(0x7fb0b00a9b60 0x7fb0b01430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.511+0000 7fb0b5f39640 1 -- 172.21.15.106:0/2421351087 >> 172.21.15.106:0/2421351087 conn(0x7fb0b00b7ec0 msgr2=0x7fb0b00b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.511+0000 7fb0b5f39640 1 -- 172.21.15.106:0/2421351087 shutdown_connections 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.511+0000 7fb0b5f39640 1 -- 172.21.15.106:0/2421351087 wait complete. 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.511+0000 7fb0b5f39640 10 monclient: build_initial_monmap 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.511+0000 7fb0b5f39640 10 monclient: monmap: 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.512+0000 7fb0b5f39640 1 librados: starting msgr at 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.512+0000 7fb0b5f39640 1 librados: starting objecter 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.512+0000 7fb0b5f39640 1 Processor -- start 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.512+0000 7fb0b5f39640 1 -- start start 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.512+0000 7fb0b5f39640 1 librados: setting wanted keys 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.512+0000 7fb0b5f39640 1 librados: calling monclient init 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.512+0000 7fb0b5f39640 10 monclient: init 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.512+0000 7fb0b5f39640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:25.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.512+0000 7fb0b5f39640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0b5f39640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0b00a9b60 0x7fb0b0145070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0b5f39640 10 monclient(hunting): picked mon.noname-c con 0x7fb0b00a9b60 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0b5f39640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b013f810 0x7fb0b0144660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0b5f39640 10 monclient(hunting): picked mon.noname-b con 0x7fb0b013f810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0b5f39640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0b01db180 0x7fb0b01db610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0b5f39640 10 monclient(hunting): picked mon.noname-a con 0x7fb0b01db180 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0b5f39640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0b5f39640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b01401e0 con 0x7fb0b01db180 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0b5f39640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0b5f39640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b00b23b0 con 0x7fb0b013f810 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0b5f39640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0b5f39640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b0144ba0 con 0x7fb0b00a9b60 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0b5f39640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0b5f39640 10 monclient(hunting): authenticate will time out at 1394.780249s 2024-04-04T21:02:25.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0af7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0b00a9b60 0x7fb0b0145070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0affff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0b01db180 0x7fb0b01db610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0aeffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b013f810 0x7fb0b0144660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0af7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0b00a9b60 0x7fb0b0145070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:60914/0 (socket says 172.21.15.106:60914) 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0af7fe640 1 -- 172.21.15.106:0/4092779775 learned_addr learned my addr 172.21.15.106:0/4092779775 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0af7fe640 10 monclient(hunting): get_auth_request con 0x7fb0b00a9b60 auth_method 0 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0af7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0affff640 10 monclient(hunting): get_auth_request con 0x7fb0b01db180 auth_method 0 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0affff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.513+0000 7fb0affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.514+0000 7fb0aeffd640 10 monclient(hunting): get_auth_request con 0x7fb0b013f810 auth_method 0 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.514+0000 7fb0aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:25.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.514+0000 7fb0aeffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.514+0000 7fb0aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.514+0000 7fb0af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.514+0000 7fb0af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.514+0000 7fb0af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.514+0000 7fb0affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.514+0000 7fb0affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.514+0000 7fb0affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.514+0000 7fb0aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.514+0000 7fb0aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.514+0000 7fb0aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.517+0000 7fb0af7fe640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.517+0000 7fb0af7fe640 1 -- 172.21.15.106:0/4092779775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b013f810 msgr2=0x7fb0b0144660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.517+0000 7fb0af7fe640 1 --2- 172.21.15.106:0/4092779775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b013f810 0x7fb0b0144660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.517+0000 7fb0af7fe640 1 -- 172.21.15.106:0/4092779775 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0b01db180 msgr2=0x7fb0b01db610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:25.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.517+0000 7fb0af7fe640 1 --2- 172.21.15.106:0/4092779775 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0b01db180 0x7fb0b01db610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:25.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.517+0000 7fb0af7fe640 10 monclient: _finish_hunting 0 2024-04-04T21:02:25.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.517+0000 7fb0af7fe640 1 monclient: found mon.noname-c 2024-04-04T21:02:25.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.517+0000 7fb0af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:25.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.517+0000 7fb0af7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:02:25.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.517+0000 7fb0af7fe640 1 -- 172.21.15.106:0/4092779775 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b01dac80 con 0x7fb0b00a9b60 2024-04-04T21:02:25.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.519+0000 7fb0af7fe640 1 --2- 172.21.15.106:0/4092779775 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0b00a9b60 0x7fb0b0145070 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000e910 tx=0x7fb0a000edd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:25.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.519+0000 7fb097fff640 10 client.?.objecter ms_handle_connect 0x7fb0b00a9b60 2024-04-04T21:02:25.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.519+0000 7fb097fff640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:25.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb097fff640 1 -- 172.21.15.106:0/4092779775 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0017070 con 0x7fb0b00a9b60 2024-04-04T21:02:25.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb097fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:25.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb097fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:25.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb097fff640 10 monclient: dump: 2024-04-04T21:02:25.531 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:25.531 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:25.531 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:25.531 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:25.531 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:25.531 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:25.531 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:25.531 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:25.531 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:25.531 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:25.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb097fff640 10 monclient: _finish_auth 0 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb097fff640 10 monclient: _check_auth_tickets 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb097fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb097fff640 1 -- 172.21.15.106:0/4092779775 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a000ca00 con 0x7fb0b00a9b60 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb097fff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb097fff640 1 -- 172.21.15.106:0/4092779775 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a000cdd0 con 0x7fb0b00a9b60 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb0b5f39640 5 monclient: authenticate success, global_id 4259 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb0b5f39640 10 monclient: _renew_subs 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb0b5f39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb0b5f39640 1 -- 172.21.15.106:0/4092779775 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b00ac7a0 con 0x7fb0b00a9b60 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb097fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb097fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb097fff640 10 monclient: dump: 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb0adffb640 4 set_mon_vals no callback set 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb0b5f39640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb0b5f39640 10 monclient: _renew_subs 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb0b5f39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb0b5f39640 1 -- 172.21.15.106:0/4092779775 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b01443b0 con 0x7fb0b00a9b60 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.522+0000 7fb0b5f39640 1 librados: init done 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.523+0000 7fb0b5f39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.523+0000 7fb0b5f39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:25.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.523+0000 7fb0b5f39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:25.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.523+0000 7fb0b5f39640 1 -- 172.21.15.106:0/4092779775 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b004f050 con 0x7fb0b00a9b60 2024-04-04T21:02:25.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.544+0000 7fb097fff640 1 -- 172.21.15.106:0/4092779775 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb0a001a070 con 0x7fb0b00a9b60 2024-04-04T21:02:25.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.544+0000 7fb097fff640 1 --2- 172.21.15.106:0/4092779775 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb09c06ce60 0x7fb09c06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:25.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.544+0000 7fb0aeffd640 1 --2- 172.21.15.106:0/4092779775 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb09c06ce60 0x7fb09c06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:25.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.544+0000 7fb0aeffd640 10 monclient: get_auth_request con 0x7fb09c06ce60 auth_method 0 2024-04-04T21:02:25.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.545+0000 7fb097fff640 10 client.4259.objecter ms_handle_connect 0x7fb09c06ce60 2024-04-04T21:02:25.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.545+0000 7fb0aeffd640 1 --2- 172.21.15.106:0/4092779775 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb09c06ce60 0x7fb09c06f320 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb0b01d5550 tx=0x7fb0a4009240 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:25.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.546+0000 7fb097fff640 1 -- 172.21.15.106:0/4092779775 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb0a0057030 con 0x7fb0b00a9b60 2024-04-04T21:02:25.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.546+0000 7fb097fff640 10 client.4259.objecter ms_dispatch 0x7fb0b0000b90 osd_map(25..25 src has 1..25) v4 2024-04-04T21:02:25.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.546+0000 7fb097fff640 3 client.4259.objecter handle_osd_map got epochs [25,25] > 0 2024-04-04T21:02:25.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.546+0000 7fb097fff640 3 client.4259.objecter handle_osd_map decoding full epoch 25 2024-04-04T21:02:25.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.546+0000 7fb097fff640 20 client.4259.objecter dump_active .. 0 homeless 2024-04-04T21:02:25.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.782+0000 7fb097fff640 1 -- 172.21.15.106:0/4092779775 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb0a0058f90 con 0x7fb0b00a9b60 2024-04-04T21:02:25.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.782+0000 7fb097fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:25.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.782+0000 7fb097fff640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:25.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.969+0000 7fb0b5f39640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:25.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.969+0000 7fb0b5f39640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:25.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.969+0000 7fb0b5f39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:25.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:25.969+0000 7fb0b5f39640 1 -- 172.21.15.106:0/4092779775 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0b0067d40 con 0x7fb0b00a9b60 2024-04-04T21:02:26.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.135+0000 7fb097fff640 1 -- 172.21.15.106:0/4092779775 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+14766 (secure 0 0 0) 0x7fb0a005d030 con 0x7fb0b00a9b60 2024-04-04T21:02:26.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.135+0000 7fb097fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:26.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.135+0000 7fb097fff640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:26.136 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:02:26.136 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":25,"fsid":"8f89ce43-1bcb-41cf-9555-c40eeb258155","created":"2024-04-04T21:01:20.057083+0000","modified":"2024-04-04T21:02:24.946482+0000","last_up_change":"2024-04-04T21:02:16.990959+0000","last_in_change":"2024-04-04T21:01:25.642687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-04T21:02:04.746208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-04T21:02:21.817409+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"28337134-5253-454a-bcda-47267b05b2c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6819","nonce":2744620362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6821","nonce":2744620362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6825","nonce":2744620362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6823","nonce":2744620362}]},"public_addr":"172.21.15.106:6819/2744620362","cluster_addr":"172.21.15.106:6821/2744620362","heartbeat_back_addr":"172.21.15.106:6825/2744620362","heartbeat_front_addr":"172.21.15.106:6823/2744620362","state":["exists","up"]},{"osd":1,"uuid":"ae336a0e-2a90-4323-9f0f-1701fffe6b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6827","nonce":2321665741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6829","nonce":2321665741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6833","nonce":2321665741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6831","nonce":2321665741}]},"public_addr":"172.21.15.106:6827/2321665741","cluster_addr":"172.21.15.106:6829/2321665741","heartbeat_back_addr":"172.21.15.106:6833/2321665741","heartbeat_front_addr":"172.21.15.106:6831/2321665741","state":["exists","up"]},{"osd":2,"uuid":"773087e0-78c9-4d14-bf0a-fb605c21f92f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6811","nonce":459340769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6813","nonce":459340769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6817","nonce":459340769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6815","nonce":459340769}]},"public_addr":"172.21.15.106:6811/459340769","cluster_addr":"172.21.15.106:6813/459340769","heartbeat_back_addr":"172.21.15.106:6817/459340769","heartbeat_front_addr":"172.21.15.106:6815/459340769","state":["exists","up"]},{"osd":3,"uuid":"07da9513-c746-46d1-995f-6f8e91338b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6803","nonce":873500775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6805","nonce":873500775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6809","nonce":873500775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6807","nonce":873500775}]},"public_addr":"172.21.15.106:6803/873500775","cluster_addr":"172.21.15.106:6805/873500775","heartbeat_back_addr":"172.21.15.106:6809/873500775","heartbeat_front_addr":"172.21.15.106:6807/873500775","state":["exists","up"]},{"osd":4,"uuid":"c47d034c-1db6-4332-bf24-722fb2846808","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6825","nonce":2225997929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6827","nonce":2225997929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6831","nonce":2225997929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6829","nonce":2225997929}]},"public_addr":"172.21.15.203:6825/2225997929","cluster_addr":"172.21.15.203:6827/2225997929","heartbeat_back_addr":"172.21.15.203:6831/2225997929","heartbeat_front_addr":"172.21.15.203:6829/2225997929","state":["exists","up"]},{"osd":5,"uuid":"588caa62-e773-44ae-94e5-ece54fe980d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6801","nonce":696438187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6803","nonce":696438187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6807","nonce":696438187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6805","nonce":696438187}]},"public_addr":"172.21.15.203:6801/696438187","cluster_addr":"172.21.15.203:6803/696438187","heartbeat_back_addr":"172.21.15.203:6807/696438187","heartbeat_front_addr":"172.21.15.203:6805/696438187","state":["exists","up"]},{"osd":6,"uuid":"ef776e12-7c80-4095-b173-5b2d24996f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6817","nonce":2758852830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6819","nonce":2758852830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6823","nonce":2758852830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6821","nonce":2758852830}]},"public_addr":"172.21.15.203:6817/2758852830","cluster_addr":"172.21.15.203:6819/2758852830","heartbeat_back_addr":"172.21.15.203:6823/2758852830","heartbeat_front_addr":"172.21.15.203:6821/2758852830","state":["exists","up"]},{"osd":7,"uuid":"53aafc58-6463-4aeb-8395-e14fcfcee5de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6809","nonce":4104640840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6811","nonce":4104640840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6815","nonce":4104640840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6813","nonce":4104640840}]},"public_addr":"172.21.15.203:6809/4104640840","cluster_addr":"172.21.15.203:6811/4104640840","heartbeat_back_addr":"172.21.15.203:6815/4104640840","heartbeat_front_addr":"172.21.15.203:6813/4104640840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:03.982282+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:05.073358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:04.386488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:04.994186+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:02.522349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:01.591956+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:02.315592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:01.851115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-04T21:02:26.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 10 librados: watch_flush enter 2024-04-04T21:02:26.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 10 librados: watch_flush exit 2024-04-04T21:02:26.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 20 client.4259.objecter shutdown clearing up homeless session... 2024-04-04T21:02:26.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 10 client.4259.objecter successfully canceled tick 2024-04-04T21:02:26.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 1 -- 172.21.15.106:0/4092779775 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb09c06ce60 msgr2=0x7fb09c06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:26.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 1 --2- 172.21.15.106:0/4092779775 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb09c06ce60 0x7fb09c06f320 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb0b01d5550 tx=0x7fb0a4009240 comp rx=0 tx=0).stop 2024-04-04T21:02:26.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 10 monclient: shutdown 2024-04-04T21:02:26.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:26.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 1 -- 172.21.15.106:0/4092779775 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0b00a9b60 msgr2=0x7fb0b0145070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:26.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 1 --2- 172.21.15.106:0/4092779775 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0b00a9b60 0x7fb0b0145070 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000e910 tx=0x7fb0a000edd0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 1 -- 172.21.15.106:0/4092779775 shutdown_connections 2024-04-04T21:02:26.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 1 --2- 172.21.15.106:0/4092779775 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb09c06ce60 0x7fb09c06f320 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 1 --2- 172.21.15.106:0/4092779775 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0b01db180 0x7fb0b01db610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 1 --2- 172.21.15.106:0/4092779775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b013f810 0x7fb0b0144660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 1 --2- 172.21.15.106:0/4092779775 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0b00a9b60 0x7fb0b0145070 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 1 -- 172.21.15.106:0/4092779775 >> 172.21.15.106:0/4092779775 conn(0x7fb0b00b7ec0 msgr2=0x7fb0b00b5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:26.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 1 -- 172.21.15.106:0/4092779775 shutdown_connections 2024-04-04T21:02:26.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.137+0000 7fb0b5f39640 1 -- 172.21.15.106:0/4092779775 wait complete. 2024-04-04T21:02:26.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.138+0000 7fb0b5f39640 1 librados: shutdown 2024-04-04T21:02:26.149 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-04T21:02:26.149 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-04T21:02:26.250 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.250+0000 7f2e82d21640 10 monclient: get_monmap_and_config 2024-04-04T21:02:26.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.250+0000 7f2e82d21640 10 monclient: build_initial_monmap 2024-04-04T21:02:26.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.250+0000 7f2e82d21640 10 monclient: monmap: 2024-04-04T21:02:26.251 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:26.251 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.251 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:26.251234+0000 2024-04-04T21:02:26.251 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:26.251234+0000 2024-04-04T21:02:26.251 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:26.251 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.251 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:26.252 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:26.252 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:26.252 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:26.252 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:26.252 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.250+0000 7f2e82d21640 1 Processor -- start 2024-04-04T21:02:26.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.250+0000 7f2e82d21640 1 -- start start 2024-04-04T21:02:26.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.250+0000 7f2e82d21640 10 monclient: init 2024-04-04T21:02:26.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:26.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-04T21:02:26.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2e7c0a9b60 0x7f2e7c1430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 10 monclient(hunting): picked mon.noname-c con 0x7f2e7c0a9b60 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:26.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2e7c1426f0 0x7f2e7c0b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 10 monclient(hunting): picked mon.noname-a con 0x7f2e7c1426f0 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:26.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e7c13f810 0x7f2e7c13fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 10 monclient(hunting): picked mon.noname-b con 0x7f2e7c13f810 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:26.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7c0a9f40 con 0x7f2e7c1426f0 2024-04-04T21:02:26.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7c142c10 con 0x7f2e7c0a9b60 2024-04-04T21:02:26.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c1435e0 con 0x7f2e7c13f810 2024-04-04T21:02:26.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:26.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.251+0000 7f2e82d21640 10 monclient(hunting): authenticate will time out at 1395.518696s 2024-04-04T21:02:26.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e7bfff640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2e7c1426f0 0x7f2e7c0b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e812be640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e7c13f810 0x7f2e7c13fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e80abd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2e7c0a9b60 0x7f2e7c1430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e7bfff640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2e7c1426f0 0x7f2e7c0b28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34738/0 (socket says 172.21.15.106:34738) 2024-04-04T21:02:26.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e7bfff640 1 -- 172.21.15.106:0/3725472379 learned_addr learned my addr 172.21.15.106:0/3725472379 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:26.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e7bfff640 10 monclient(hunting): get_auth_request con 0x7f2e7c1426f0 auth_method 0 2024-04-04T21:02:26.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e7bfff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e812be640 10 monclient(hunting): get_auth_request con 0x7f2e7c13f810 auth_method 0 2024-04-04T21:02:26.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e812be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e812be640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e812be640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e80abd640 10 monclient(hunting): get_auth_request con 0x7f2e7c0a9b60 auth_method 0 2024-04-04T21:02:26.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e80abd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e80abd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e80abd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.252+0000 7f2e7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.253+0000 7f2e812be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.253+0000 7f2e812be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.253+0000 7f2e812be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.253+0000 7f2e80abd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.253+0000 7f2e80abd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.253+0000 7f2e80abd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.254+0000 7f2e7bfff640 10 monclient(hunting): handle_auth_done global_id 4287 payload 274 2024-04-04T21:02:26.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.254+0000 7f2e7bfff640 1 -- 172.21.15.106:0/3725472379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e7c13f810 msgr2=0x7f2e7c13fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.254+0000 7f2e7bfff640 1 --2- 172.21.15.106:0/3725472379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e7c13f810 0x7f2e7c13fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.254+0000 7f2e7bfff640 1 -- 172.21.15.106:0/3725472379 >> v2:172.21.15.203:3300/0 conn(0x7f2e7c0a9b60 msgr2=0x7f2e7c1430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.254+0000 7f2e7bfff640 1 --2- 172.21.15.106:0/3725472379 >> v2:172.21.15.203:3300/0 conn(0x7f2e7c0a9b60 0x7f2e7c1430a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.254+0000 7f2e7bfff640 10 monclient: _finish_hunting 0 2024-04-04T21:02:26.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.254+0000 7f2e7bfff640 1 monclient: found mon.noname-a 2024-04-04T21:02:26.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.254+0000 7f2e7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:26.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.254+0000 7f2e7bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:26.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.254+0000 7f2e7bfff640 1 -- 172.21.15.106:0/3725472379 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c0b2e00 con 0x7f2e7c1426f0 2024-04-04T21:02:26.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.256+0000 7f2e7bfff640 1 --2- 172.21.15.106:0/3725472379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e7c1426f0 0x7f2e7c0b28c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c0b59d0 tx=0x7f2e6402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbc198b7571ed953 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:26.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e7b7fe640 1 -- 172.21.15.106:0/3725472379 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6402e070 con 0x7f2e7c1426f0 2024-04-04T21:02:26.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:26.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e7b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:26.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e7b7fe640 10 monclient: dump: 2024-04-04T21:02:26.266 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:26.267 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.267 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.267 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.267 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:26.267 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.267 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:26.267 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:26.267 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:26.267 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e7b7fe640 10 monclient: _finish_auth 0 2024-04-04T21:02:26.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e7b7fe640 10 monclient: _check_auth_tickets 2024-04-04T21:02:26.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:26.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e7b7fe640 1 -- 172.21.15.106:0/3725472379 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e64037070 con 0x7f2e7c1426f0 2024-04-04T21:02:26.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:26.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e7b7fe640 1 -- 172.21.15.106:0/3725472379 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e64031a00 con 0x7f2e7c1426f0 2024-04-04T21:02:26.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e82d21640 5 monclient: authenticate success, global_id 4287 2024-04-04T21:02:26.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e82d21640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:26.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e82d21640 4 set_mon_vals no callback set 2024-04-04T21:02:26.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:26.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e7b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:26.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.260+0000 7f2e7b7fe640 10 monclient: dump: 2024-04-04T21:02:26.268 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:26.268 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.268 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.268 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.268 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:26.268 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.268 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:26.269 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:26.269 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:26.269 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.261+0000 7f2e82d21640 10 monclient: shutdown 2024-04-04T21:02:26.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.261+0000 7f2e82d21640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:26.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.261+0000 7f2e82d21640 1 -- 172.21.15.106:0/3725472379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e7c1426f0 msgr2=0x7f2e7c0b28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:26.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.261+0000 7f2e82d21640 1 --2- 172.21.15.106:0/3725472379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e7c1426f0 0x7f2e7c0b28c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c0b59d0 tx=0x7f2e6402b880 comp rx=0 tx=0).stop 2024-04-04T21:02:26.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.261+0000 7f2e82d21640 1 -- 172.21.15.106:0/3725472379 shutdown_connections 2024-04-04T21:02:26.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.261+0000 7f2e82d21640 1 --2- 172.21.15.106:0/3725472379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e7c13f810 0x7f2e7c13fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.261+0000 7f2e82d21640 1 --2- 172.21.15.106:0/3725472379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e7c1426f0 0x7f2e7c0b28c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.261+0000 7f2e82d21640 1 --2- 172.21.15.106:0/3725472379 >> v2:172.21.15.203:3300/0 conn(0x7f2e7c0a9b60 0x7f2e7c1430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.261+0000 7f2e82d21640 1 -- 172.21.15.106:0/3725472379 >> 172.21.15.106:0/3725472379 conn(0x7f2e7c0b7ec0 msgr2=0x7f2e7c0b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:26.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.261+0000 7f2e82d21640 1 -- 172.21.15.106:0/3725472379 shutdown_connections 2024-04-04T21:02:26.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.261+0000 7f2e82d21640 1 -- 172.21.15.106:0/3725472379 wait complete. 2024-04-04T21:02:26.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.261+0000 7f2e82d21640 10 monclient: build_initial_monmap 2024-04-04T21:02:26.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.261+0000 7f2e82d21640 10 monclient: monmap: 2024-04-04T21:02:26.270 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:26.270 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:26.270 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:26.270 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:26.270 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:26.270 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.270 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:26.270 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:26.271 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:26.271 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 1 librados: starting msgr at 2024-04-04T21:02:26.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 1 librados: starting objecter 2024-04-04T21:02:26.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 1 Processor -- start 2024-04-04T21:02:26.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 1 -- start start 2024-04-04T21:02:26.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 1 librados: setting wanted keys 2024-04-04T21:02:26.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 1 librados: calling monclient init 2024-04-04T21:02:26.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 10 monclient: init 2024-04-04T21:02:26.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:26.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-04T21:02:26.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e7c0a9b60 0x7f2e7c1daff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 10 monclient(hunting): picked mon.noname-b con 0x7f2e7c0a9b60 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:26.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e7c13f810 0x7f2e7c13d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 10 monclient(hunting): picked mon.noname-c con 0x7f2e7c13f810 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:26.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e7c1dba30 0x7f2e7c135d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 10 monclient(hunting): picked mon.noname-a con 0x7f2e7c1dba30 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:26.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c1401e0 con 0x7f2e7c1dba30 2024-04-04T21:02:26.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c0b23b0 con 0x7f2e7c0a9b60 2024-04-04T21:02:26.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c0ac980 con 0x7f2e7c13f810 2024-04-04T21:02:26.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:26.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.262+0000 7f2e82d21640 10 monclient(hunting): authenticate will time out at 1395.530171s 2024-04-04T21:02:26.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.263+0000 7f2e80abd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e7c0a9b60 0x7f2e7c1daff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.263+0000 7f2e80abd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e7c0a9b60 0x7f2e7c1daff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50064/0 (socket says 172.21.15.106:50064) 2024-04-04T21:02:26.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.263+0000 7f2e80abd640 1 -- 172.21.15.106:0/516621538 learned_addr learned my addr 172.21.15.106:0/516621538 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.263+0000 7f2e80abd640 10 monclient(hunting): get_auth_request con 0x7f2e7c0a9b60 auth_method 0 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.263+0000 7f2e80abd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.263+0000 7f2e80abd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.263+0000 7f2e80abd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.264+0000 7f2e812be640 1 --2- 172.21.15.106:0/516621538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e7c1dba30 0x7f2e7c135d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.264+0000 7f2e80abd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.264+0000 7f2e80abd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.264+0000 7f2e80abd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.264+0000 7f2e812be640 10 monclient(hunting): get_auth_request con 0x7f2e7c1dba30 auth_method 0 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.264+0000 7f2e812be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.264+0000 7f2e812be640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.264+0000 7f2e812be640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.265+0000 7f2e812be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.265+0000 7f2e812be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.265+0000 7f2e812be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.265+0000 7f2e7bfff640 1 --2- 172.21.15.106:0/516621538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e7c13f810 0x7f2e7c13d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e7bfff640 10 monclient(hunting): get_auth_request con 0x7f2e7c13f810 auth_method 0 2024-04-04T21:02:26.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e7bfff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.276 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.276 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.276 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.276 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e80abd640 10 monclient(hunting): handle_auth_done global_id 4306 payload 824 2024-04-04T21:02:26.276 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e80abd640 1 -- 172.21.15.106:0/516621538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e7c13f810 msgr2=0x7f2e7c13d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:26.276 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e80abd640 1 --2- 172.21.15.106:0/516621538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e7c13f810 0x7f2e7c13d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e80abd640 1 -- 172.21.15.106:0/516621538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e7c1dba30 msgr2=0x7f2e7c135d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:26.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e80abd640 1 --2- 172.21.15.106:0/516621538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e7c1dba30 0x7f2e7c135d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e80abd640 10 monclient: _finish_hunting 0 2024-04-04T21:02:26.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e80abd640 1 monclient: found mon.noname-b 2024-04-04T21:02:26.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e80abd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:26.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e80abd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:26.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.266+0000 7f2e80abd640 1 -- 172.21.15.106:0/516621538 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c0acc60 con 0x7f2e7c0a9b60 2024-04-04T21:02:26.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.269+0000 7f2e80abd640 1 --2- 172.21.15.106:0/516621538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e7c0a9b60 0x7f2e7c1daff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400eed0 tx=0x7f2e7400b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:26.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.269+0000 7f2e797fa640 10 client.?.objecter ms_handle_connect 0x7f2e7c0a9b60 2024-04-04T21:02:26.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.269+0000 7f2e797fa640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:26.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e797fa640 1 -- 172.21.15.106:0/516621538 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e74018020 con 0x7f2e7c0a9b60 2024-04-04T21:02:26.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:26.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e797fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:26.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e797fa640 10 monclient: dump: 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e797fa640 10 monclient: _finish_auth 0 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e797fa640 10 monclient: _check_auth_tickets 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e797fa640 1 -- 172.21.15.106:0/516621538 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7400bd90 con 0x7f2e7c0a9b60 2024-04-04T21:02:26.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e797fa640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:26.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e797fa640 1 -- 172.21.15.106:0/516621538 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e74014620 con 0x7f2e7c0a9b60 2024-04-04T21:02:26.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:26.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:26.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e797fa640 10 monclient: dump: 2024-04-04T21:02:26.279 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:26.279 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.279 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.279 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.279 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:26.279 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.279 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:26.279 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:26.279 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:26.279 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e82d21640 5 monclient: authenticate success, global_id 4306 2024-04-04T21:02:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e82d21640 10 monclient: _renew_subs 2024-04-04T21:02:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e82d21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e7affd640 4 set_mon_vals no callback set 2024-04-04T21:02:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e82d21640 1 -- 172.21.15.106:0/516621538 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7c13dd60 con 0x7f2e7c0a9b60 2024-04-04T21:02:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e82d21640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e82d21640 10 monclient: _renew_subs 2024-04-04T21:02:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e82d21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e82d21640 1 -- 172.21.15.106:0/516621538 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7c136260 con 0x7f2e7c0a9b60 2024-04-04T21:02:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.275+0000 7f2e82d21640 1 librados: init done 2024-04-04T21:02:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.276+0000 7f2e82d21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.276+0000 7f2e82d21640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.276+0000 7f2e82d21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.276+0000 7f2e82d21640 1 -- 172.21.15.106:0/516621538 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7c04f050 con 0x7f2e7c0a9b60 2024-04-04T21:02:26.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.523+0000 7f2e797fa640 1 -- 172.21.15.106:0/516621538 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2e74005d10 con 0x7f2e7c0a9b60 2024-04-04T21:02:26.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.523+0000 7f2e797fa640 1 --2- 172.21.15.106:0/516621538 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2e5c06ce60 0x7f2e5c06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.523+0000 7f2e7bfff640 1 --2- 172.21.15.106:0/516621538 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2e5c06ce60 0x7f2e5c06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.523+0000 7f2e7bfff640 10 monclient: get_auth_request con 0x7f2e5c06ce60 auth_method 0 2024-04-04T21:02:26.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.524+0000 7f2e797fa640 10 client.4306.objecter ms_handle_connect 0x7f2e5c06ce60 2024-04-04T21:02:26.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.524+0000 7f2e7bfff640 1 --2- 172.21.15.106:0/516621538 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2e5c06ce60 0x7f2e5c06f320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2e64036000 tx=0x7f2e6403d040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:26.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.525+0000 7f2e797fa640 1 -- 172.21.15.106:0/516621538 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2e74057980 con 0x7f2e7c0a9b60 2024-04-04T21:02:26.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.525+0000 7f2e797fa640 10 client.4306.objecter ms_dispatch 0x7f2e7c000b90 osd_map(26..26 src has 1..26) v4 2024-04-04T21:02:26.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.525+0000 7f2e797fa640 3 client.4306.objecter handle_osd_map got epochs [26,26] > 0 2024-04-04T21:02:26.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.525+0000 7f2e797fa640 3 client.4306.objecter handle_osd_map decoding full epoch 26 2024-04-04T21:02:26.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.525+0000 7f2e797fa640 20 client.4306.objecter dump_active .. 0 homeless 2024-04-04T21:02:26.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.535+0000 7f2e797fa640 1 -- 172.21.15.106:0/516621538 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2e74022070 con 0x7f2e7c0a9b60 2024-04-04T21:02:26.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.535+0000 7f2e797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:26.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.535+0000 7f2e797fa640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:26.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.716+0000 7f2e82d21640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:26.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.716+0000 7f2e82d21640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:26.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.716+0000 7f2e82d21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:26.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.716+0000 7f2e82d21640 1 -- 172.21.15.106:0/516621538 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e7c067d40 con 0x7f2e7c0a9b60 2024-04-04T21:02:26.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.749+0000 7f2e797fa640 1 -- 172.21.15.106:0/516621538 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+14753 (secure 0 0 0) 0x7f2e740598e0 con 0x7f2e7c0a9b60 2024-04-04T21:02:26.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.749+0000 7f2e797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:02:26.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.749+0000 7f2e797fa640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:26.749 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:02:26.750 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":26,"fsid":"8f89ce43-1bcb-41cf-9555-c40eeb258155","created":"2024-04-04T21:01:20.057083+0000","modified":"2024-04-04T21:02:26.034392+0000","last_up_change":"2024-04-04T21:02:16.990959+0000","last_in_change":"2024-04-04T21:01:25.642687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-04T21:02:04.746208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-04T21:02:21.817409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"28337134-5253-454a-bcda-47267b05b2c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6819","nonce":2744620362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6821","nonce":2744620362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6825","nonce":2744620362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6823","nonce":2744620362}]},"public_addr":"172.21.15.106:6819/2744620362","cluster_addr":"172.21.15.106:6821/2744620362","heartbeat_back_addr":"172.21.15.106:6825/2744620362","heartbeat_front_addr":"172.21.15.106:6823/2744620362","state":["exists","up"]},{"osd":1,"uuid":"ae336a0e-2a90-4323-9f0f-1701fffe6b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6827","nonce":2321665741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6829","nonce":2321665741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6833","nonce":2321665741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6831","nonce":2321665741}]},"public_addr":"172.21.15.106:6827/2321665741","cluster_addr":"172.21.15.106:6829/2321665741","heartbeat_back_addr":"172.21.15.106:6833/2321665741","heartbeat_front_addr":"172.21.15.106:6831/2321665741","state":["exists","up"]},{"osd":2,"uuid":"773087e0-78c9-4d14-bf0a-fb605c21f92f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6811","nonce":459340769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6813","nonce":459340769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6817","nonce":459340769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6815","nonce":459340769}]},"public_addr":"172.21.15.106:6811/459340769","cluster_addr":"172.21.15.106:6813/459340769","heartbeat_back_addr":"172.21.15.106:6817/459340769","heartbeat_front_addr":"172.21.15.106:6815/459340769","state":["exists","up"]},{"osd":3,"uuid":"07da9513-c746-46d1-995f-6f8e91338b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6803","nonce":873500775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6805","nonce":873500775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6809","nonce":873500775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6807","nonce":873500775}]},"public_addr":"172.21.15.106:6803/873500775","cluster_addr":"172.21.15.106:6805/873500775","heartbeat_back_addr":"172.21.15.106:6809/873500775","heartbeat_front_addr":"172.21.15.106:6807/873500775","state":["exists","up"]},{"osd":4,"uuid":"c47d034c-1db6-4332-bf24-722fb2846808","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6825","nonce":2225997929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6827","nonce":2225997929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6831","nonce":2225997929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6829","nonce":2225997929}]},"public_addr":"172.21.15.203:6825/2225997929","cluster_addr":"172.21.15.203:6827/2225997929","heartbeat_back_addr":"172.21.15.203:6831/2225997929","heartbeat_front_addr":"172.21.15.203:6829/2225997929","state":["exists","up"]},{"osd":5,"uuid":"588caa62-e773-44ae-94e5-ece54fe980d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6801","nonce":696438187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6803","nonce":696438187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6807","nonce":696438187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6805","nonce":696438187}]},"public_addr":"172.21.15.203:6801/696438187","cluster_addr":"172.21.15.203:6803/696438187","heartbeat_back_addr":"172.21.15.203:6807/696438187","heartbeat_front_addr":"172.21.15.203:6805/696438187","state":["exists","up"]},{"osd":6,"uuid":"ef776e12-7c80-4095-b173-5b2d24996f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6817","nonce":2758852830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6819","nonce":2758852830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6823","nonce":2758852830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6821","nonce":2758852830}]},"public_addr":"172.21.15.203:6817/2758852830","cluster_addr":"172.21.15.203:6819/2758852830","heartbeat_back_addr":"172.21.15.203:6823/2758852830","heartbeat_front_addr":"172.21.15.203:6821/2758852830","state":["exists","up"]},{"osd":7,"uuid":"53aafc58-6463-4aeb-8395-e14fcfcee5de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6809","nonce":4104640840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6811","nonce":4104640840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6815","nonce":4104640840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6813","nonce":4104640840}]},"public_addr":"172.21.15.203:6809/4104640840","cluster_addr":"172.21.15.203:6811/4104640840","heartbeat_back_addr":"172.21.15.203:6815/4104640840","heartbeat_front_addr":"172.21.15.203:6813/4104640840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:03.982282+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:05.073358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:04.386488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:04.994186+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:02.522349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:01.591956+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:02.315592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:01.851115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-04T21:02:26.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.750+0000 7f2e82d21640 10 librados: watch_flush enter 2024-04-04T21:02:26.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.750+0000 7f2e82d21640 10 librados: watch_flush exit 2024-04-04T21:02:26.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.750+0000 7f2e82d21640 20 client.4306.objecter shutdown clearing up homeless session... 2024-04-04T21:02:26.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.750+0000 7f2e82d21640 10 client.4306.objecter successfully canceled tick 2024-04-04T21:02:26.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.750+0000 7f2e82d21640 1 -- 172.21.15.106:0/516621538 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2e5c06ce60 msgr2=0x7f2e5c06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:26.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.750+0000 7f2e82d21640 1 --2- 172.21.15.106:0/516621538 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2e5c06ce60 0x7f2e5c06f320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2e64036000 tx=0x7f2e6403d040 comp rx=0 tx=0).stop 2024-04-04T21:02:26.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.750+0000 7f2e82d21640 10 monclient: shutdown 2024-04-04T21:02:26.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.750+0000 7f2e82d21640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:26.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.750+0000 7f2e82d21640 1 -- 172.21.15.106:0/516621538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e7c0a9b60 msgr2=0x7f2e7c1daff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:26.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.750+0000 7f2e82d21640 1 --2- 172.21.15.106:0/516621538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e7c0a9b60 0x7f2e7c1daff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400eed0 tx=0x7f2e7400b480 comp rx=0 tx=0).stop 2024-04-04T21:02:26.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.751+0000 7f2e82d21640 1 -- 172.21.15.106:0/516621538 shutdown_connections 2024-04-04T21:02:26.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.751+0000 7f2e82d21640 1 --2- 172.21.15.106:0/516621538 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2e5c06ce60 0x7f2e5c06f320 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.751+0000 7f2e82d21640 1 --2- 172.21.15.106:0/516621538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e7c1dba30 0x7f2e7c135d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.751+0000 7f2e82d21640 1 --2- 172.21.15.106:0/516621538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e7c13f810 0x7f2e7c13d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.751+0000 7f2e82d21640 1 --2- 172.21.15.106:0/516621538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e7c0a9b60 0x7f2e7c1daff0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.751+0000 7f2e82d21640 1 -- 172.21.15.106:0/516621538 >> 172.21.15.106:0/516621538 conn(0x7f2e7c0b7ec0 msgr2=0x7f2e7c0b82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:26.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.751+0000 7f2e82d21640 1 -- 172.21.15.106:0/516621538 shutdown_connections 2024-04-04T21:02:26.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.751+0000 7f2e82d21640 1 -- 172.21.15.106:0/516621538 wait complete. 2024-04-04T21:02:26.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.751+0000 7f2e82d21640 1 librados: shutdown 2024-04-04T21:02:26.762 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-04T21:02:26.763 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-04T21:02:26.763 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-04T21:02:26.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-04T21:02:26.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-04T21:02:26.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-04T21:02:26.765 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-04T21:02:26.765 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-04T21:02:26.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.931+0000 7f77183f1640 10 monclient: get_monmap_and_config 2024-04-04T21:02:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.931+0000 7f77183f1640 10 monclient: build_initial_monmap 2024-04-04T21:02:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.931+0000 7f77183f1640 10 monclient: monmap: 2024-04-04T21:02:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:26.932676+0000 2024-04-04T21:02:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:26.932676+0000 2024-04-04T21:02:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:26.933 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.932+0000 7f0404b1c640 10 monclient: get_monmap_and_config 2024-04-04T21:02:26.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.932+0000 7f0404b1c640 10 monclient: build_initial_monmap 2024-04-04T21:02:26.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.932+0000 7f0404b1c640 10 monclient: monmap: 2024-04-04T21:02:26.934 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:26.934 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.934 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:26.933544+0000 2024-04-04T21:02:26.935 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:26.933544+0000 2024-04-04T21:02:26.935 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:26.935 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.935 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:26.935 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:26.935 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:26.935 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:26.935 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:26.935 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.936 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.932+0000 7f0d4b59e640 10 monclient: get_monmap_and_config 2024-04-04T21:02:26.936 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.932+0000 7f0d4b59e640 10 monclient: build_initial_monmap 2024-04-04T21:02:26.936 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.932+0000 7f0d4b59e640 10 monclient: monmap: 2024-04-04T21:02:26.936 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:26.936 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.936 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:26.933696+0000 2024-04-04T21:02:26.936 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:26.933696+0000 2024-04-04T21:02:26.936 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:26.936 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.936 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:26.936 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:26.936 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:26.936 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:26.937 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:26.937 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7f77183f1640 1 Processor -- start 2024-04-04T21:02:26.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7f77183f1640 1 -- start start 2024-04-04T21:02:26.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7f77183f1640 10 monclient: init 2024-04-04T21:02:26.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7f0d4b59e640 1 Processor -- start 2024-04-04T21:02:26.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7f0d4b59e640 1 -- start start 2024-04-04T21:02:26.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7f0d4b59e640 10 monclient: init 2024-04-04T21:02:26.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7f62e5da4640 10 monclient: get_monmap_and_config 2024-04-04T21:02:26.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7f62e5da4640 10 monclient: build_initial_monmap 2024-04-04T21:02:26.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7f62e5da4640 10 monclient: monmap: 2024-04-04T21:02:26.938 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:26.938 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.938 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:26.934586+0000 2024-04-04T21:02:26.938 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:26.934586+0000 2024-04-04T21:02:26.938 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:26.938 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.939 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:26.939 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:26.939 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:26.939 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:26.939 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:26.939 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7fae9a34f640 10 monclient: get_monmap_and_config 2024-04-04T21:02:26.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7fae9a34f640 10 monclient: build_initial_monmap 2024-04-04T21:02:26.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7fae9a34f640 10 monclient: monmap: 2024-04-04T21:02:26.939 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:26.939 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.940 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:26.934447+0000 2024-04-04T21:02:26.940 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:26.934447+0000 2024-04-04T21:02:26.940 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:26.940 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.940 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:26.940 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:26.940 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:26.940 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:26.940 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:26.940 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7f3a09be3640 10 monclient: get_monmap_and_config 2024-04-04T21:02:26.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7f3a09be3640 10 monclient: build_initial_monmap 2024-04-04T21:02:26.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.933+0000 7f3a09be3640 10 monclient: monmap: 2024-04-04T21:02:26.941 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:26.941 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.941 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:26.934630+0000 2024-04-04T21:02:26.941 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:26.934630+0000 2024-04-04T21:02:26.941 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:26.941 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.941 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:26.941 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:26.941 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:26.941 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:26.941 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:26.941 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f77183f1640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:26.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f77183f1640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-04-04T21:02:26.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f77183f1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f770802de60 0x7f7708032a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f77183f1640 10 monclient(hunting): picked mon.noname-c con 0x7f770802de60 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:26.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f77183f1640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770802e240 0x7f77080c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f77183f1640 10 monclient(hunting): picked mon.noname-b con 0x7f770802e240 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:26.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f77183f1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f77080c82b0 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:26.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f77183f1640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77080c5870 con 0x7f77080c82b0 2024-04-04T21:02:26.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f77183f1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f77183f1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f770802f5f0 con 0x7f770802de60 2024-04-04T21:02:26.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f77183f1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f77183f1640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f770802f8d0 con 0x7f770802e240 2024-04-04T21:02:26.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f77183f1640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:26.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f77183f1640 10 monclient(hunting): authenticate will time out at 1396.201976s 2024-04-04T21:02:26.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0d4b59e640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:26.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0d4b59e640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-04-04T21:02:26.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0d4b59e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d4c06e5a0 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:26.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0d4b59e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0d4c06f800 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:26.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0d4b59e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0d4c14a8b0 0x7f0d4c152160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0d4b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f0d4c14a8b0 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:26.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0d4b59e640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c14acc0 con 0x7f0d4c06f800 2024-04-04T21:02:26.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0d4b59e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c06f3b0 con 0x7f0d4c06e5a0 2024-04-04T21:02:26.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0d4b59e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0d4b59e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d4c06ede0 con 0x7f0d4c14a8b0 2024-04-04T21:02:26.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0d4b59e640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:26.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0d4b59e640 10 monclient(hunting): authenticate will time out at 1396.201909s 2024-04-04T21:02:26.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f62e5da4640 1 Processor -- start 2024-04-04T21:02:26.944 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7fae9a34f640 1 Processor -- start 2024-04-04T21:02:26.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f3a09be3640 1 Processor -- start 2024-04-04T21:02:26.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f19a359e640 10 monclient: get_monmap_and_config 2024-04-04T21:02:26.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f19a359e640 10 monclient: build_initial_monmap 2024-04-04T21:02:26.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f19a359e640 10 monclient: monmap: 2024-04-04T21:02:26.945 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:26.945 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.945 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:26.936163+0000 2024-04-04T21:02:26.945 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:26.936163+0000 2024-04-04T21:02:26.945 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:26.946 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.946 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:26.946 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:26.946 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:26.946 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:26.946 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:26.946 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0404b1c640 1 Processor -- start 2024-04-04T21:02:26.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0404b1c640 1 -- start start 2024-04-04T21:02:26.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7fae9a34f640 1 -- start start 2024-04-04T21:02:26.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7fae9a34f640 10 monclient: init 2024-04-04T21:02:26.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f0404b1c640 10 monclient: init 2024-04-04T21:02:26.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f771618d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f770802de60 0x7f7708032a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f0404b1c640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:26.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f0404b1c640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-04-04T21:02:26.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f0404b1c640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f04001442f0 0x7f04000b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f0404b1c640 10 monclient(hunting): picked mon.noname-a con 0x7f04001442f0 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:26.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f0404b1c640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f040014a060 0x7f04000b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f0404b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f040014a060 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:26.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f0404b1c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0400147150 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:26.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f0404b1c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040006f6a0 con 0x7f0400147150 2024-04-04T21:02:26.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f0404b1c640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f0404b1c640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f040006e130 con 0x7f04001442f0 2024-04-04T21:02:26.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f0404b1c640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f0404b1c640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0400146c30 con 0x7f040014a060 2024-04-04T21:02:26.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f771618d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f770802de60 0x7f7708032a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50080/0 (socket says 172.21.15.106:50080) 2024-04-04T21:02:26.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f0404b1c640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:26.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f0404b1c640 10 monclient(hunting): authenticate will time out at 1396.202677s 2024-04-04T21:02:26.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f771618d640 1 -- 172.21.15.106:0/3764770092 learned_addr learned my addr 172.21.15.106:0/3764770092 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:26.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7fae9a34f640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:26.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7fae9a34f640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-04-04T21:02:26.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7fae9a34f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae8c007400 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:26.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7fae9a34f640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae8c008b70 0x7fae8c0cf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7fae9a34f640 10 monclient(hunting): picked mon.noname-b con 0x7fae8c008b70 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:26.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7fae9a34f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fae8c0cfb70 0x7fae8c0d6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7fae9a34f640 10 monclient(hunting): picked mon.noname-c con 0x7fae8c0cfb70 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:26.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7fae9a34f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c0c9d40 con 0x7fae8c007400 2024-04-04T21:02:26.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7fae9a34f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7fae9a34f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8c0077e0 con 0x7fae8c0cfb70 2024-04-04T21:02:26.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7fae9a34f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7fae9a34f640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c02c8b0 con 0x7fae8c008b70 2024-04-04T21:02:26.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7fae9a34f640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:26.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7fae9a34f640 10 monclient(hunting): authenticate will time out at 1396.203011s 2024-04-04T21:02:26.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f3a09be3640 1 -- start start 2024-04-04T21:02:26.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f771618d640 10 monclient(hunting): get_auth_request con 0x7f770802de60 auth_method 0 2024-04-04T21:02:26.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f771618d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f771618d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f771618d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f0d4ad9d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0d4c14a8b0 0x7f0d4c152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f771598c640 1 --2- 172.21.15.106:0/3764770092 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770802e240 0x7f77080c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f19a359e640 1 Processor -- start 2024-04-04T21:02:26.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f62e5da4640 1 -- start start 2024-04-04T21:02:26.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f62e5da4640 10 monclient: init 2024-04-04T21:02:26.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f0d4a59c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f0d4c06e5a0 0x7f0d4c143c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:41834/0 (socket says 172.21.15.106:41834) 2024-04-04T21:02:26.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f0d4a59c640 1 -- 172.21.15.106:0/472581219 learned_addr learned my addr 172.21.15.106:0/472581219 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:26.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.934+0000 7f3a09be3640 10 monclient: init 2024-04-04T21:02:26.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f19a359e640 1 -- start start 2024-04-04T21:02:26.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f19a359e640 10 monclient: init 2024-04-04T21:02:26.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f03fdd9b640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f040014a060 0x7f04000b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f3a09be3640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:26.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f0d4ad9d640 10 monclient(hunting): get_auth_request con 0x7f0d4c14a8b0 auth_method 0 2024-04-04T21:02:26.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f0d4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f0d4ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f0d4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.957 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f3a09be3640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-04-04T21:02:26.957 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f03fdd9b640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f040014a060 0x7f04000b1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:60942/0 (socket says 172.21.15.106:60942) 2024-04-04T21:02:26.957 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f03fdd9b640 1 -- 172.21.15.106:0/3104635334 learned_addr learned my addr 172.21.15.106:0/3104635334 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:26.957 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f03fdd9b640 10 monclient(hunting): get_auth_request con 0x7f040014a060 auth_method 0 2024-04-04T21:02:26.957 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f03fdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.957 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f03fdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.957 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f03fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f3a09be3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a040b19d0 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:26.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f3a09be3640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3a040b2140 0x7f3a041402a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f771618d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f3a09be3640 10 monclient(hunting): picked mon.noname-c con 0x7f3a040b2140 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:26.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f771618d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.936+0000 7f771618d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f3a09be3640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a040afa20 0x7f3a0413d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f3a09be3640 10 monclient(hunting): picked mon.noname-b con 0x7f3a040afa20 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:26.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f3a09be3640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0406e5d0 con 0x7f3a040b19d0 2024-04-04T21:02:26.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f3a09be3640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f3a09be3640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a040b1db0 con 0x7f3a040b2140 2024-04-04T21:02:26.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f3a09be3640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f3a09be3640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0406e7a0 con 0x7f3a040afa20 2024-04-04T21:02:26.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f62e5da4640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:26.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f62e5da4640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-04-04T21:02:26.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f62e5da4640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f62e006ca40 0x7f62e013f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.961 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f3a09be3640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:26.961 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.935+0000 7f3a09be3640 10 monclient(hunting): authenticate will time out at 1396.203009s 2024-04-04T21:02:26.961 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f03fdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.961 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f03fdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.961 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f03fdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f62e5da4640 10 monclient(hunting): picked mon.noname-a con 0x7f62e006ca40 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:26.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f62e5da4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62e013fd10 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:26.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f62e5da4640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e0143000 0x7f62e014ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f62e5da4640 10 monclient(hunting): picked mon.noname-b con 0x7f62e0143000 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:26.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:26.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-04-04T21:02:26.963 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f19a4070840 0x7f19a413d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.963 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 10 monclient(hunting): picked mon.noname-a con 0x7f19a4070840 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:26.963 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19a4070c20 0x7f19a413da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.963 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f62e5da4640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e00b0300 con 0x7f62e013fd10 2024-04-04T21:02:26.963 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f62e5da4640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.963 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f62e5da4640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62e006ce20 con 0x7f62e006ca40 2024-04-04T21:02:26.963 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f62e5da4640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.963 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 10 monclient(hunting): picked mon.noname-b con 0x7f19a4070c20 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:26.964 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f19a4143af0 0x7f19a4143f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.964 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 10 monclient(hunting): picked mon.noname-c con 0x7f19a4143af0 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:26.964 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.964 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a40afdc0 con 0x7f19a4070840 2024-04-04T21:02:26.964 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.964 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a413df40 con 0x7f19a4143af0 2024-04-04T21:02:26.964 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.964 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a40727b0 con 0x7f19a4070c20 2024-04-04T21:02:26.964 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:26.964 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f62e5da4640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e0143490 con 0x7f62e0143000 2024-04-04T21:02:26.964 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f62e5da4640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:26.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f62e5da4640 10 monclient(hunting): authenticate will time out at 1396.204780s 2024-04-04T21:02:26.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f19a359e640 10 monclient(hunting): authenticate will time out at 1396.204762s 2024-04-04T21:02:26.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f0d4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f0d4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f0d4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f771598c640 10 monclient(hunting): get_auth_request con 0x7f770802e240 auth_method 0 2024-04-04T21:02:26.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f771598c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f771598c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.937+0000 7f771598c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.938+0000 7f77153ee640 10 monclient: get_monmap_and_config 2024-04-04T21:02:26.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.938+0000 7f77153ee640 10 monclient: build_initial_monmap 2024-04-04T21:02:26.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.938+0000 7f77153ee640 10 monclient: monmap: 2024-04-04T21:02:26.967 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:26.967 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.967 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:26.939830+0000 2024-04-04T21:02:26.967 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:26.939830+0000 2024-04-04T21:02:26.967 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:26.967 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.967 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:26.967 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:26.967 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:26.967 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:26.967 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:26.967 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.968 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f771618d640 10 monclient(hunting): handle_auth_done global_id 4309 payload 274 2024-04-04T21:02:26.968 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f771618d640 1 -- 172.21.15.106:0/3764770092 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770802e240 msgr2=0x7f77080c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.968 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f77153ee640 1 Processor -- start 2024-04-04T21:02:26.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f771618d640 1 --2- 172.21.15.106:0/3764770092 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770802e240 0x7f77080c7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f771618d640 1 -- 172.21.15.106:0/3764770092 >> v1:172.21.15.106:6789/0 conn(0x7f77080c82b0 legacy=0x7f77080cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:26.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f771618d640 10 monclient: _finish_hunting 0 2024-04-04T21:02:26.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f771618d640 1 monclient: found mon.noname-c 2024-04-04T21:02:26.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f771618d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:26.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f771618d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:02:26.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f771618d640 1 -- 172.21.15.106:0/3764770092 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770802fbb0 con 0x7f770802de60 2024-04-04T21:02:26.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f03fe59c640 1 --2- 172.21.15.106:0/3104635334 >> v2:172.21.15.106:3300/0 conn(0x7f04001442f0 0x7f04000b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f03fdd9b640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-04-04T21:02:26.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f03fdd9b640 1 -- 172.21.15.106:0/3104635334 >> v2:172.21.15.106:3300/0 conn(0x7f04001442f0 msgr2=0x7f04000b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f03fdd9b640 1 --2- 172.21.15.106:0/3104635334 >> v2:172.21.15.106:3300/0 conn(0x7f04001442f0 0x7f04000b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f03fdd9b640 1 -- 172.21.15.106:0/3104635334 >> v1:172.21.15.203:6789/0 conn(0x7f0400147150 legacy=0x7f04000b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f03fdd9b640 10 monclient: _finish_hunting 0 2024-04-04T21:02:26.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f03fdd9b640 1 monclient: found mon.noname-b 2024-04-04T21:02:26.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f77153ee640 1 -- start start 2024-04-04T21:02:26.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.939+0000 7f77153ee640 10 monclient: init 2024-04-04T21:02:26.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f03fdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:26.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f03fdd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:26.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f03fdd9b640 1 -- 172.21.15.106:0/3104635334 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04000b2170 con 0x7f040014a060 2024-04-04T21:02:26.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f77153ee640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:26.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f77153ee640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-04-04T21:02:26.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f77153ee640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f77080071c0 0x7f7708030820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:26.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f77153ee640 10 monclient(hunting): picked mon.noname-a con 0x7f77080071c0 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:26.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f77153ee640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f77080cfaa0 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:26.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f77153ee640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7708033550 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:26.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f77153ee640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77080375e0 con 0x7f77080cfaa0 2024-04-04T21:02:26.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f77153ee640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77080cf580 con 0x7f7708033550 2024-04-04T21:02:26.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f77153ee640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:26.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f77153ee640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7708035120 con 0x7f77080071c0 2024-04-04T21:02:26.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f77153ee640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:26.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.940+0000 7f77153ee640 10 monclient(hunting): authenticate will time out at 1396.207723s 2024-04-04T21:02:26.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.942+0000 7f0d4ad9d640 10 monclient(hunting): handle_auth_done global_id 4312 payload 274 2024-04-04T21:02:26.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.942+0000 7f0d4ad9d640 1 -- 172.21.15.106:0/472581219 >> v1:172.21.15.203:6789/0 conn(0x7f0d4c06e5a0 legacy=0x7f0d4c143c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:26.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.942+0000 7f0d4ad9d640 1 -- 172.21.15.106:0/472581219 >> v1:172.21.15.106:6789/0 conn(0x7f0d4c06f800 legacy=0x7f0d4c14a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:26.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.942+0000 7f0d4ad9d640 10 monclient: _finish_hunting 0 2024-04-04T21:02:26.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.942+0000 7f0d4ad9d640 1 monclient: found mon.noname-c 2024-04-04T21:02:26.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.942+0000 7f0d4ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:26.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.942+0000 7f0d4ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:02:26.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.942+0000 7f0d4ad9d640 1 -- 172.21.15.106:0/472581219 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c070f60 con 0x7f0d4c14a8b0 2024-04-04T21:02:26.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.943+0000 7f3a093e2640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a040afa20 0x7f3a0413d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f770ffff640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f77080071c0 0x7f7708030820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f3a093e2640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a040afa20 0x7f3a0413d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:60960/0 (socket says 172.21.15.106:60960) 2024-04-04T21:02:26.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f3a093e2640 1 -- 172.21.15.106:0/1299049164 learned_addr learned my addr 172.21.15.106:0/1299049164 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:26.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f3a093e2640 10 monclient(hunting): get_auth_request con 0x7f3a040afa20 auth_method 0 2024-04-04T21:02:26.975 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f3a093e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.975 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f3a093e2640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.975 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f3a093e2640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.975 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f770ffff640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f77080071c0 0x7f7708030820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34788/0 (socket says 172.21.15.106:34788) 2024-04-04T21:02:26.975 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f770ffff640 1 -- 172.21.15.106:0/1850960981 learned_addr learned my addr 172.21.15.106:0/1850960981 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:26.975 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f770ffff640 10 monclient(hunting): get_auth_request con 0x7f77080071c0 auth_method 0 2024-04-04T21:02:26.975 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f770ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.975 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f770ffff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.975 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f770ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f3a093e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f3a093e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f3a093e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.944+0000 7f19a259c640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f19a4070840 0x7f19a413d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.945+0000 7f62df7fe640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f62e006ca40 0x7f62e013f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.946+0000 7f770ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.946+0000 7f770ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.946+0000 7f770ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.946+0000 7f19a259c640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f19a4070840 0x7f19a413d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34778/0 (socket says 172.21.15.106:34778) 2024-04-04T21:02:26.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.946+0000 7f19a259c640 1 -- 172.21.15.106:0/381212499 learned_addr learned my addr 172.21.15.106:0/381212499 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:26.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.946+0000 7f19a259c640 10 monclient(hunting): get_auth_request con 0x7f19a4070840 auth_method 0 2024-04-04T21:02:26.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.946+0000 7f19a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.946+0000 7f19a259c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.946+0000 7f19a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.947+0000 7f62df7fe640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f62e006ca40 0x7f62e013f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34766/0 (socket says 172.21.15.106:34766) 2024-04-04T21:02:26.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.947+0000 7f62df7fe640 1 -- 172.21.15.106:0/1774802058 learned_addr learned my addr 172.21.15.106:0/1774802058 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:26.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.947+0000 7f62df7fe640 10 monclient(hunting): get_auth_request con 0x7f62e006ca40 auth_method 0 2024-04-04T21:02:26.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.947+0000 7f62df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.947+0000 7f62df7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.947+0000 7f62df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.947+0000 7f771618d640 1 --2- 172.21.15.106:0/3764770092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f770802de60 0x7f7708032a40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7710050830 tx=0x7f77100a6720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c933fe47c44c14a server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:26.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.947+0000 7f19a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.947+0000 7f19a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.947+0000 7f19a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.948+0000 7f3a093e2640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-04-04T21:02:26.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.948+0000 7f3a093e2640 1 -- 172.21.15.106:0/1299049164 >> v2:172.21.15.203:3300/0 conn(0x7f3a040b2140 msgr2=0x7f3a041402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.948+0000 7f3a093e2640 1 --2- 172.21.15.106:0/1299049164 >> v2:172.21.15.203:3300/0 conn(0x7f3a040b2140 0x7f3a041402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.948+0000 7f62df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.948+0000 7f62df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.948+0000 7f62df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.948+0000 7f3a093e2640 1 -- 172.21.15.106:0/1299049164 >> v1:172.21.15.106:6789/0 conn(0x7f3a040b19d0 legacy=0x7f3a04071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.948+0000 7f3a093e2640 10 monclient: _finish_hunting 0 2024-04-04T21:02:26.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.948+0000 7f3a093e2640 1 monclient: found mon.noname-b 2024-04-04T21:02:26.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.948+0000 7f3a093e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:26.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.948+0000 7f3a093e2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:26.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.948+0000 7f3a093e2640 1 -- 172.21.15.106:0/1299049164 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0413da00 con 0x7f3a040afa20 2024-04-04T21:02:26.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.948+0000 7f03fdd9b640 1 --2- 172.21.15.106:0/3104635334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f040014a060 0x7f04000b1c30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f03f8009e50 tx=0x7f03f802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=468c8319cbb540d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:26.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7fae99b4e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fae8c0cfb70 0x7fae8c0d6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7fae98b4c640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae8c008b70 0x7fae8c0cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7f770ffff640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2024-04-04T21:02:26.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7f770ffff640 1 -- 172.21.15.106:0/1850960981 >> v1:172.21.15.203:6789/0 conn(0x7f7708033550 legacy=0x7f77080339a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7f770ffff640 1 -- 172.21.15.106:0/1850960981 >> v1:172.21.15.106:6789/0 conn(0x7f77080cfaa0 legacy=0x7f77080cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7f770ffff640 10 monclient: _finish_hunting 0 2024-04-04T21:02:26.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7f770ffff640 1 monclient: found mon.noname-a 2024-04-04T21:02:26.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7f770ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:26.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7f770ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:26.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7f770ffff640 1 -- 172.21.15.106:0/1850960981 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7708038da0 con 0x7f77080071c0 2024-04-04T21:02:26.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7fae99b4e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fae8c0cfb70 0x7fae8c0d6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50102/0 (socket says 172.21.15.106:50102) 2024-04-04T21:02:26.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7fae99b4e640 1 -- 172.21.15.106:0/3980829455 learned_addr learned my addr 172.21.15.106:0/3980829455 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:26.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7fae99b4e640 10 monclient(hunting): get_auth_request con 0x7fae8c0cfb70 auth_method 0 2024-04-04T21:02:26.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7fae99b4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7fae99b4e640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.949+0000 7fae99b4e640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.950+0000 7f62dffff640 1 --2- 172.21.15.106:0/1774802058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e0143000 0x7f62e014ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.951+0000 7f62dffff640 10 monclient(hunting): get_auth_request con 0x7f62e0143000 auth_method 0 2024-04-04T21:02:26.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.951+0000 7f62dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.951+0000 7f62dffff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.951+0000 7f62dffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.952+0000 7fae98b4c640 10 monclient(hunting): get_auth_request con 0x7fae8c008b70 auth_method 0 2024-04-04T21:02:26.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.952+0000 7fae98b4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.952+0000 7fae98b4c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.952+0000 7fae98b4c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.952+0000 7f62df7fe640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2024-04-04T21:02:26.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.952+0000 7f62df7fe640 1 -- 172.21.15.106:0/1774802058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e0143000 msgr2=0x7f62e014ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.952+0000 7f62df7fe640 1 --2- 172.21.15.106:0/1774802058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e0143000 0x7f62e014ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.952+0000 7f62df7fe640 1 -- 172.21.15.106:0/1774802058 >> v1:172.21.15.106:6789/0 conn(0x7f62e013fd10 legacy=0x7f62e01428f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.952+0000 7f62df7fe640 10 monclient: _finish_hunting 0 2024-04-04T21:02:26.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.952+0000 7f62df7fe640 1 monclient: found mon.noname-a 2024-04-04T21:02:26.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.952+0000 7f62df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:26.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.952+0000 7f62df7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:26.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.952+0000 7f62df7fe640 1 -- 172.21.15.106:0/1774802058 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e00ae4e0 con 0x7f62e006ca40 2024-04-04T21:02:26.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.952+0000 7f3a093e2640 1 --2- 172.21.15.106:0/1299049164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a040afa20 0x7f3a0413d4c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f39ec009870 tx=0x7f39ec02b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53e4651e1fc4f16 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:26.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.954+0000 7fae98b4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.954+0000 7fae98b4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.954+0000 7fae98b4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.955+0000 7f770ffff640 1 --2- 172.21.15.106:0/1850960981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080071c0 0x7f7708030820 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7704009850 tx=0x7f770402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec7556f1656c7afe server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:26.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.956+0000 7fae99b4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.956+0000 7fae99b4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.956+0000 7fae99b4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.956+0000 7f0d4ad9d640 1 --2- 172.21.15.106:0/472581219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d4c14a8b0 0x7f0d4c152160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0d400097a0 tx=0x7f0d4002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57021f123c84d2b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:26.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.956+0000 7f62df7fe640 1 --2- 172.21.15.106:0/1774802058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62e006ca40 0x7f62e013f7d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009870 tx=0x7f62d402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0153abc2333dfbd server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:26.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.957+0000 7fae98b4c640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2024-04-04T21:02:26.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.957+0000 7fae98b4c640 1 -- 172.21.15.106:0/3980829455 >> v2:172.21.15.203:3300/0 conn(0x7fae8c0cfb70 msgr2=0x7fae8c0d6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.957+0000 7fae98b4c640 1 --2- 172.21.15.106:0/3980829455 >> v2:172.21.15.203:3300/0 conn(0x7fae8c0cfb70 0x7fae8c0d6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.957+0000 7fae98b4c640 1 -- 172.21.15.106:0/3980829455 >> v1:172.21.15.203:6789/0 conn(0x7fae8c007400 legacy=0x7fae8c0c9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.957+0000 7fae98b4c640 10 monclient: _finish_hunting 0 2024-04-04T21:02:26.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.957+0000 7fae98b4c640 1 monclient: found mon.noname-b 2024-04-04T21:02:26.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.957+0000 7fae98b4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:26.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.957+0000 7fae98b4c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:26.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.957+0000 7fae98b4c640 1 -- 172.21.15.106:0/3980829455 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c009120 con 0x7fae8c008b70 2024-04-04T21:02:26.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.957+0000 7f19a2d9d640 1 --2- 172.21.15.106:0/381212499 >> v2:172.21.15.203:3300/0 conn(0x7f19a4143af0 0x7f19a4143f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:26.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.958+0000 7f19a2d9d640 10 monclient(hunting): get_auth_request con 0x7f19a4143af0 auth_method 0 2024-04-04T21:02:26.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.958+0000 7f19a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:26.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.958+0000 7f19a2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:26.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.958+0000 7f19a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:26.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.961+0000 7f19a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:26.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.961+0000 7f19a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:26.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.961+0000 7f19a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:26.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.961+0000 7f19a259c640 10 monclient(hunting): handle_auth_done global_id 4302 payload 274 2024-04-04T21:02:26.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.961+0000 7f19a259c640 1 -- 172.21.15.106:0/381212499 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19a4070c20 msgr2=0x7f19a413da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.961+0000 7f19a259c640 1 --2- 172.21.15.106:0/381212499 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19a4070c20 0x7f19a413da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.961+0000 7f19a259c640 1 -- 172.21.15.106:0/381212499 >> v2:172.21.15.203:3300/0 conn(0x7f19a4143af0 msgr2=0x7f19a4143f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:26.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.961+0000 7f19a259c640 1 --2- 172.21.15.106:0/381212499 >> v2:172.21.15.203:3300/0 conn(0x7f19a4143af0 0x7f19a4143f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:26.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.961+0000 7f19a259c640 10 monclient: _finish_hunting 0 2024-04-04T21:02:26.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.961+0000 7f19a259c640 1 monclient: found mon.noname-a 2024-04-04T21:02:26.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.961+0000 7f19a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:26.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.961+0000 7f19a259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:26.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.961+0000 7f19a259c640 1 -- 172.21.15.106:0/381212499 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a4072a90 con 0x7f19a4070840 2024-04-04T21:02:26.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a037fe640 1 -- 172.21.15.106:0/1299049164 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec038070 con 0x7f3a040afa20 2024-04-04T21:02:26.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:26.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:26.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a037fe640 10 monclient: dump: 2024-04-04T21:02:26.994 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:26.994 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.994 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.994 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.994 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:26.994 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.995 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:26.995 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:26.995 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:26.995 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a037fe640 10 monclient: _finish_auth 0 2024-04-04T21:02:26.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a037fe640 10 monclient: _check_auth_tickets 2024-04-04T21:02:26.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:26.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a09be3640 5 monclient: authenticate success, global_id 4274 2024-04-04T21:02:26.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a09be3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:26.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a037fe640 1 -- 172.21.15.106:0/1299049164 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ec031740 con 0x7f3a040afa20 2024-04-04T21:02:26.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a037fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:26.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a09be3640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:26.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a09be3640 4 set_mon_vals no callback set 2024-04-04T21:02:26.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a037fe640 1 -- 172.21.15.106:0/1299049164 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec038070 con 0x7f3a040afa20 2024-04-04T21:02:26.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:26.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:26.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f3a037fe640 10 monclient: dump: 2024-04-04T21:02:26.996 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:26.996 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.996 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.996 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.996 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:26.996 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.996 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:26.996 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:26.996 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:26.996 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.962+0000 7f771518b640 1 -- 172.21.15.106:0/3764770092 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100b3070 con 0x7f770802de60 2024-04-04T21:02:26.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f771518b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:26.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f771518b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:26.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f771518b640 10 monclient: dump: 2024-04-04T21:02:26.997 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:26.997 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.997 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.997 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.998 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:26.998 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.998 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:26.998 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:26.998 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:26.998 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f771518b640 10 monclient: _finish_auth 0 2024-04-04T21:02:26.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f771518b640 10 monclient: _check_auth_tickets 2024-04-04T21:02:26.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f771518b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:26.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f771518b640 1 -- 172.21.15.106:0/3764770092 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77100ab770 con 0x7f770802de60 2024-04-04T21:02:26.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f771518b640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:26.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f771518b640 1 -- 172.21.15.106:0/3764770092 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100abbe0 con 0x7f770802de60 2024-04-04T21:02:26.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f771518b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:26.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f771518b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:26.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f771518b640 10 monclient: dump: 2024-04-04T21:02:26.999 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:26.999 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:26.999 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.999 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:26.999 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:26.999 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:26.999 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:26.999 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:26.999 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:26.999 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:26.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f77183f1640 5 monclient: authenticate success, global_id 4309 2024-04-04T21:02:26.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f77183f1640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:26.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f77183f1640 4 set_mon_vals no callback set 2024-04-04T21:02:27.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f3a09be3640 10 monclient: shutdown 2024-04-04T21:02:27.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f3a09be3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f3a09be3640 1 -- 172.21.15.106:0/1299049164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a040afa20 msgr2=0x7f3a0413d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f3a09be3640 1 --2- 172.21.15.106:0/1299049164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a040afa20 0x7f3a0413d4c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f39ec009870 tx=0x7f39ec02b7e0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f3a09be3640 1 -- 172.21.15.106:0/1299049164 shutdown_connections 2024-04-04T21:02:27.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f3a09be3640 1 --2- 172.21.15.106:0/1299049164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a040afa20 0x7f3a0413d4c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f3a09be3640 1 --2- 172.21.15.106:0/1299049164 >> v2:172.21.15.203:3300/0 conn(0x7f3a040b2140 0x7f3a041402a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f3a09be3640 1 -- 172.21.15.106:0/1299049164 >> 172.21.15.106:0/1299049164 conn(0x7f3a040b2850 msgr2=0x7f3a040b2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f77183f1640 10 monclient: shutdown 2024-04-04T21:02:27.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f77183f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f77183f1640 1 -- 172.21.15.106:0/3764770092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f770802de60 msgr2=0x7f7708032a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f77183f1640 1 --2- 172.21.15.106:0/3764770092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f770802de60 0x7f7708032a40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7710050830 tx=0x7f77100a6720 comp rx=0 tx=0).stop 2024-04-04T21:02:27.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f03fd59a640 1 -- 172.21.15.106:0/3104635334 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f802c9e0 con 0x7f040014a060 2024-04-04T21:02:27.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f03fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f03fd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:27.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f03fd59a640 10 monclient: dump: 2024-04-04T21:02:27.002 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.002 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.002 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.002 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.002 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.002 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.003 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.003 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.003 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.003 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f03fd59a640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f3a09be3640 1 -- 172.21.15.106:0/1299049164 shutdown_connections 2024-04-04T21:02:27.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f03fd59a640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f03fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f0404b1c640 5 monclient: authenticate success, global_id 4268 2024-04-04T21:02:27.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f03fd59a640 1 -- 172.21.15.106:0/3104635334 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f802cb80 con 0x7f040014a060 2024-04-04T21:02:27.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f0404b1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:27.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f03fd59a640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f0404b1c640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:27.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f0404b1c640 4 set_mon_vals no callback set 2024-04-04T21:02:27.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f03fd59a640 1 -- 172.21.15.106:0/3104635334 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f80343f0 con 0x7f040014a060 2024-04-04T21:02:27.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f03fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f03fd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:27.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.963+0000 7f03fd59a640 10 monclient: dump: 2024-04-04T21:02:27.004 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.005 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.005 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.005 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.005 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.005 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.005 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.005 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.005 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.005 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f77183f1640 1 -- 172.21.15.106:0/3764770092 shutdown_connections 2024-04-04T21:02:27.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f77183f1640 1 --2- 172.21.15.106:0/3764770092 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770802e240 0x7f77080c7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f77183f1640 1 --2- 172.21.15.106:0/3764770092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f770802de60 0x7f7708032a40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f77183f1640 1 -- 172.21.15.106:0/3764770092 >> 172.21.15.106:0/3764770092 conn(0x7f7708036610 msgr2=0x7f7708005e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f0404b1c640 10 monclient: shutdown 2024-04-04T21:02:27.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f0404b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f0404b1c640 1 -- 172.21.15.106:0/3104635334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f040014a060 msgr2=0x7f04000b1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f0404b1c640 1 --2- 172.21.15.106:0/3104635334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f040014a060 0x7f04000b1c30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f03f8009e50 tx=0x7f03f802be40 comp rx=0 tx=0).stop 2024-04-04T21:02:27.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f3a09be3640 1 -- 172.21.15.106:0/1299049164 wait complete. 2024-04-04T21:02:27.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f77183f1640 1 -- 172.21.15.106:0/3764770092 shutdown_connections 2024-04-04T21:02:27.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f0404b1c640 1 -- 172.21.15.106:0/3104635334 shutdown_connections 2024-04-04T21:02:27.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f0404b1c640 1 --2- 172.21.15.106:0/3104635334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f040014a060 0x7f04000b1c30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f0404b1c640 1 --2- 172.21.15.106:0/3104635334 >> v2:172.21.15.106:3300/0 conn(0x7f04001442f0 0x7f04000b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f0404b1c640 1 -- 172.21.15.106:0/3104635334 >> 172.21.15.106:0/3104635334 conn(0x7f04000b8030 msgr2=0x7f04000b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f77183f1640 1 -- 172.21.15.106:0/3764770092 wait complete. 2024-04-04T21:02:27.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f0404b1c640 1 -- 172.21.15.106:0/3104635334 shutdown_connections 2024-04-04T21:02:27.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f0404b1c640 1 -- 172.21.15.106:0/3104635334 wait complete. 2024-04-04T21:02:27.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f3a09be3640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.964+0000 7f3a09be3640 10 monclient: monmap: 2024-04-04T21:02:27.009 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.009 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.009 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.009 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.010 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.010 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.010 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.010 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.010 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.010 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.965+0000 7f3a09be3640 1 librados: starting msgr at 2024-04-04T21:02:27.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.965+0000 7f3a09be3640 1 librados: starting objecter 2024-04-04T21:02:27.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.965+0000 7f3a09be3640 1 Processor -- start 2024-04-04T21:02:27.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.965+0000 7fae98b4c640 1 --2- 172.21.15.106:0/3980829455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae8c008b70 0x7fae8c0cf630 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fae94074350 tx=0x7fae940a65d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed9a8f93c629c0fe server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.965+0000 7f3a09be3640 1 -- start start 2024-04-04T21:02:27.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.965+0000 7f3a09be3640 1 librados: setting wanted keys 2024-04-04T21:02:27.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.965+0000 7f3a09be3640 1 librados: calling monclient init 2024-04-04T21:02:27.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.965+0000 7f3a09be3640 10 monclient: init 2024-04-04T21:02:27.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.965+0000 7f77183f1640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.965+0000 7f0404b1c640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.965+0000 7f0404b1c640 10 monclient: monmap: 2024-04-04T21:02:27.012 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.012 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.012 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.012 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.012 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.012 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.012 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.012 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.012 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.012 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.965+0000 7f77183f1640 10 monclient: monmap: 2024-04-04T21:02:27.013 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.013 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.013 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.013 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.013 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.013 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.013 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.013 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.013 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.013 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.965+0000 7f0404b1c640 1 librados: starting msgr at 2024-04-04T21:02:27.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.965+0000 7f0404b1c640 1 librados: starting objecter 2024-04-04T21:02:27.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 1 Processor -- start 2024-04-04T21:02:27.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 1 -- start start 2024-04-04T21:02:27.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 1 librados: setting wanted keys 2024-04-04T21:02:27.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 1 librados: calling monclient init 2024-04-04T21:02:27.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 10 monclient: init 2024-04-04T21:02:27.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-04T21:02:27.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a040afa20 0x7f3a04149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f77183f1640 1 librados: starting msgr at 2024-04-04T21:02:27.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f77183f1640 1 librados: starting objecter 2024-04-04T21:02:27.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 10 monclient(hunting): picked mon.noname-a con 0x7f3a040afa20 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a040b19d0 0x7f3a0413f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 10 monclient(hunting): picked mon.noname-b con 0x7f3a040b19d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a04143de0 0x7f3a04143280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 10 monclient(hunting): picked mon.noname-c con 0x7f3a04143de0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0414ab30 con 0x7f3a040afa20 2024-04-04T21:02:27.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f77183f1640 1 Processor -- start 2024-04-04T21:02:27.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a040712e0 con 0x7f3a040b19d0 2024-04-04T21:02:27.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0414acb0 con 0x7f3a04143de0 2024-04-04T21:02:27.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f3a09be3640 10 monclient(hunting): authenticate will time out at 1396.233638s 2024-04-04T21:02:27.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f77183f1640 1 -- start start 2024-04-04T21:02:27.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f77183f1640 1 librados: setting wanted keys 2024-04-04T21:02:27.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f77183f1640 1 librados: calling monclient init 2024-04-04T21:02:27.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f77183f1640 10 monclient: init 2024-04-04T21:02:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-04T21:02:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04001442f0 0x7f04001df800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f04001442f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0400147150 0x7f0400120cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 10 monclient(hunting): picked mon.noname-a con 0x7f0400147150 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f040014a060 0x7f0400121210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 10 monclient(hunting): picked mon.noname-c con 0x7f040014a060 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0400147970 con 0x7f0400147150 2024-04-04T21:02:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04001477f0 con 0x7f04001442f0 2024-04-04T21:02:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04000b3140 con 0x7f040014a060 2024-04-04T21:02:27.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.966+0000 7f0404b1c640 10 monclient(hunting): authenticate will time out at 1396.234026s 2024-04-04T21:02:27.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-04T21:02:27.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770802de60 0x7f77080cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 10 monclient(hunting): picked mon.noname-c con 0x7f770802de60 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080c82b0 0x7f770815d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 10 monclient(hunting): picked mon.noname-a con 0x7f77080c82b0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f770815e1d0 0x7f7708160b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 10 monclient(hunting): picked mon.noname-b con 0x7f770815e1d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770802d6f0 con 0x7f77080c82b0 2024-04-04T21:02:27.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770802c8c0 con 0x7f770815e1d0 2024-04-04T21:02:27.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7708032f80 con 0x7f770802de60 2024-04-04T21:02:27.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.026 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.967+0000 7f77183f1640 10 monclient(hunting): authenticate will time out at 1396.235148s 2024-04-04T21:02:27.026 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62de7fc640 1 -- 172.21.15.106:0/1774802058 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d4035020 con 0x7f62e006ca40 2024-04-04T21:02:27.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:27.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62de7fc640 10 monclient: dump: 2024-04-04T21:02:27.027 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.027 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.027 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.027 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.027 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.027 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.027 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.027 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.027 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.027 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62de7fc640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62de7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62de7fc640 1 -- 172.21.15.106:0/1774802058 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d4031640 con 0x7f62e006ca40 2024-04-04T21:02:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62de7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62de7fc640 1 -- 172.21.15.106:0/1774802058 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d402d040 con 0x7f62e006ca40 2024-04-04T21:02:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62e5da4640 5 monclient: authenticate success, global_id 4305 2024-04-04T21:02:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62e5da4640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62e5da4640 4 set_mon_vals no callback set 2024-04-04T21:02:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.969+0000 7f62de7fc640 10 monclient: dump: 2024-04-04T21:02:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f62e5da4640 10 monclient: shutdown 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f62e5da4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f62e5da4640 1 -- 172.21.15.106:0/1774802058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62e006ca40 msgr2=0x7f62e013f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f62e5da4640 1 --2- 172.21.15.106:0/1774802058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62e006ca40 0x7f62e013f7d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009870 tx=0x7f62d402b8f0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f62e5da4640 1 -- 172.21.15.106:0/1774802058 shutdown_connections 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f62e5da4640 1 --2- 172.21.15.106:0/1774802058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e0143000 0x7f62e014ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f62e5da4640 1 --2- 172.21.15.106:0/1774802058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62e006ca40 0x7f62e013f7d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f62e5da4640 1 -- 172.21.15.106:0/1774802058 >> 172.21.15.106:0/1774802058 conn(0x7f62e00b62c0 msgr2=0x7f62e00b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f62e5da4640 1 -- 172.21.15.106:0/1774802058 shutdown_connections 2024-04-04T21:02:27.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f62e5da4640 1 -- 172.21.15.106:0/1774802058 wait complete. 2024-04-04T21:02:27.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f0d4959a640 1 -- 172.21.15.106:0/472581219 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d40035020 con 0x7f0d4c14a8b0 2024-04-04T21:02:27.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f0d4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f0d4959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:27.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f0d4959a640 10 monclient: dump: 2024-04-04T21:02:27.031 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.031 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.031 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.031 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.031 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.031 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.031 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.031 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.031 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.031 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f0d4959a640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f0d4959a640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f0d4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f0d4b59e640 5 monclient: authenticate success, global_id 4312 2024-04-04T21:02:27.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f0d4b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:27.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f0d4959a640 1 -- 172.21.15.106:0/472581219 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d40031830 con 0x7f0d4c14a8b0 2024-04-04T21:02:27.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f0d4959a640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f0d4b59e640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:27.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.970+0000 7f0d4b59e640 4 set_mon_vals no callback set 2024-04-04T21:02:27.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f0d4959a640 1 -- 172.21.15.106:0/472581219 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d40031c00 con 0x7f0d4c14a8b0 2024-04-04T21:02:27.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f0d4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f0d4959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:27.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f3a093e2640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a04143de0 0x7f3a04143280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f0d4959a640 10 monclient: dump: 2024-04-04T21:02:27.033 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.033 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.033 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.033 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.033 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.033 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.033 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.033 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.033 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.033 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f62e5da4640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f62e5da4640 10 monclient: monmap: 2024-04-04T21:02:27.034 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.034 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.034 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.034 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.034 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.034 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.034 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.035 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.035 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.035 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f0d4b59e640 10 monclient: shutdown 2024-04-04T21:02:27.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f0d4b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f0d4b59e640 1 -- 172.21.15.106:0/472581219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d4c14a8b0 msgr2=0x7f0d4c152160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f0d4b59e640 1 --2- 172.21.15.106:0/472581219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d4c14a8b0 0x7f0d4c152160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0d400097a0 tx=0x7f0d4002b8d0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f3a093e2640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a04143de0 0x7f3a04143280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:60984/0 (socket says 172.21.15.106:60984) 2024-04-04T21:02:27.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f3a093e2640 1 -- 172.21.15.106:0/3436398859 learned_addr learned my addr 172.21.15.106:0/3436398859 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f3a093e2640 10 monclient(hunting): get_auth_request con 0x7f3a04143de0 auth_method 0 2024-04-04T21:02:27.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f3a093e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f3a093e2640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f3a093e2640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f03fed9d640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f040014a060 0x7f0400121210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f0d4b59e640 1 -- 172.21.15.106:0/472581219 shutdown_connections 2024-04-04T21:02:27.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f0d4b59e640 1 --2- 172.21.15.106:0/472581219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d4c14a8b0 0x7f0d4c152160 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f0d4b59e640 1 -- 172.21.15.106:0/472581219 >> 172.21.15.106:0/472581219 conn(0x7f0d4c0b3e10 msgr2=0x7f0d4c0b4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f0d4b59e640 1 -- 172.21.15.106:0/472581219 shutdown_connections 2024-04-04T21:02:27.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f62e5da4640 1 librados: starting msgr at 2024-04-04T21:02:27.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f62e5da4640 1 librados: starting objecter 2024-04-04T21:02:27.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f0d4b59e640 1 -- 172.21.15.106:0/472581219 wait complete. 2024-04-04T21:02:27.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.971+0000 7f62e5da4640 1 Processor -- start 2024-04-04T21:02:27.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f62e5da4640 1 -- start start 2024-04-04T21:02:27.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f3a093e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f3a093e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f3a093e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f62e5da4640 1 librados: setting wanted keys 2024-04-04T21:02:27.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f62e5da4640 1 librados: calling monclient init 2024-04-04T21:02:27.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f62e5da4640 10 monclient: init 2024-04-04T21:02:27.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f3a03fff640 1 --2- 172.21.15.106:0/3436398859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a040b19d0 0x7f3a0413f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.040 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f0d4b59e640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.040 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f0d4b59e640 10 monclient: monmap: 2024-04-04T21:02:27.040 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.040 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.040 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.040 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.040 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.040 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.040 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.040 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.040 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.040 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.041 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f770effd640 1 -- 172.21.15.106:0/1850960981 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7704004230 con 0x7f77080071c0 2024-04-04T21:02:27.041 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f770effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.041 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f770effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:27.041 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f770effd640 10 monclient: dump: 2024-04-04T21:02:27.041 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f770effd640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f770effd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f770effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f770effd640 1 -- 172.21.15.106:0/1850960981 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77040043d0 con 0x7f77080071c0 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f77153ee640 5 monclient: authenticate success, global_id 4299 2024-04-04T21:02:27.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f77153ee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f770effd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f770effd640 1 -- 172.21.15.106:0/1850960981 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7704030840 con 0x7f77080071c0 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f770effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f770effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f770effd640 10 monclient: dump: 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.043 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f77153ee640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:27.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f77153ee640 4 set_mon_vals no callback set 2024-04-04T21:02:27.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f3a03fff640 10 monclient(hunting): get_auth_request con 0x7f3a040b19d0 auth_method 0 2024-04-04T21:02:27.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f3a03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f3a03fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f3a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 1 librados: starting msgr at 2024-04-04T21:02:27.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f62e5da4640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f62e5da4640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:02:27.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.972+0000 7f62e5da4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62e006ca40 0x7f62e00b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f62e5da4640 10 monclient(hunting): picked mon.noname-a con 0x7f62e006ca40 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f62e5da4640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e013fd10 0x7f62e00702a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f62e5da4640 10 monclient(hunting): picked mon.noname-c con 0x7f62e013fd10 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f62e5da4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62e0143000 0x7f62e0072b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f62e5da4640 10 monclient(hunting): picked mon.noname-b con 0x7f62e0143000 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f62e5da4640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f62e5da4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e006ddb0 con 0x7f62e006ca40 2024-04-04T21:02:27.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f62e5da4640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 1 librados: starting objecter 2024-04-04T21:02:27.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f62e5da4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e006dc30 con 0x7f62e0143000 2024-04-04T21:02:27.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f62e5da4640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f62e5da4640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e006cfd0 con 0x7f62e013fd10 2024-04-04T21:02:27.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f62e5da4640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f62e5da4640 10 monclient(hunting): authenticate will time out at 1396.240255s 2024-04-04T21:02:27.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f03fe59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04001442f0 0x7f04001df800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 1 Processor -- start 2024-04-04T21:02:27.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f03fed9d640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f040014a060 0x7f0400121210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:60994/0 (socket says 172.21.15.106:60994) 2024-04-04T21:02:27.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f03fed9d640 1 -- 172.21.15.106:0/3245008356 learned_addr learned my addr 172.21.15.106:0/3245008356 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f03fed9d640 10 monclient(hunting): get_auth_request con 0x7f040014a060 auth_method 0 2024-04-04T21:02:27.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f03fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f03fed9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f03fed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 1 -- start start 2024-04-04T21:02:27.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 1 librados: setting wanted keys 2024-04-04T21:02:27.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 1 librados: calling monclient init 2024-04-04T21:02:27.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 10 monclient: init 2024-04-04T21:02:27.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f03fe59c640 10 monclient(hunting): get_auth_request con 0x7f04001442f0 auth_method 0 2024-04-04T21:02:27.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f03fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f03fe59c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f03fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f77153ee640 10 monclient: shutdown 2024-04-04T21:02:27.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f77153ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f77153ee640 1 -- 172.21.15.106:0/1850960981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080071c0 msgr2=0x7f7708030820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f77153ee640 1 --2- 172.21.15.106:0/1850960981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080071c0 0x7f7708030820 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7704009850 tx=0x7f770402b980 comp rx=0 tx=0).stop 2024-04-04T21:02:27.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f77153ee640 1 -- 172.21.15.106:0/1850960981 shutdown_connections 2024-04-04T21:02:27.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f77153ee640 1 --2- 172.21.15.106:0/1850960981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080071c0 0x7f7708030820 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-04T21:02:27.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f77153ee640 1 -- 172.21.15.106:0/1850960981 >> 172.21.15.106:0/1850960981 conn(0x7f7708006190 msgr2=0x7f77080065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d4c06e5a0 0x7f0d4c145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f0d4c06e5a0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0d4c14a8b0 0x7f0d4c0b2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f0d4c14a8b0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0d4c1464e0 0x7f0d4c0b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f0d4c1464e0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c1444b0 con 0x7f0d4c1464e0 2024-04-04T21:02:27.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f77153ee640 1 -- 172.21.15.106:0/1850960981 shutdown_connections 2024-04-04T21:02:27.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c144330 con 0x7f0d4c06e5a0 2024-04-04T21:02:27.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c06e980 con 0x7f0d4c14a8b0 2024-04-04T21:02:27.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.973+0000 7f0d4b59e640 10 monclient(hunting): authenticate will time out at 1396.241156s 2024-04-04T21:02:27.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.974+0000 7f77153ee640 1 -- 172.21.15.106:0/1850960981 wait complete. 2024-04-04T21:02:27.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.974+0000 7f3a03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.974+0000 7f3a03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.974+0000 7f3a03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.974+0000 7f77153ee640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.974+0000 7f77153ee640 10 monclient: monmap: 2024-04-04T21:02:27.054 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.054 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.054 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.054 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.054 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.054 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.054 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.054 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.054 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.055 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.975+0000 7f77153ee640 1 librados: starting msgr at 2024-04-04T21:02:27.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.975+0000 7f77153ee640 1 librados: starting objecter 2024-04-04T21:02:27.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.975+0000 7f77153ee640 1 Processor -- start 2024-04-04T21:02:27.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.975+0000 7f77153ee640 1 -- start start 2024-04-04T21:02:27.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.975+0000 7f77153ee640 1 librados: setting wanted keys 2024-04-04T21:02:27.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.975+0000 7f77153ee640 1 librados: calling monclient init 2024-04-04T21:02:27.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.975+0000 7f77153ee640 10 monclient: init 2024-04-04T21:02:27.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.975+0000 7f77153ee640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-04T21:02:27.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77080071c0 0x7f77080c2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 10 monclient(hunting): picked mon.noname-b con 0x7f77080071c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7708033550 0x7f770802fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f19a259c640 1 --2- 172.21.15.106:0/381212499 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a4070840 0x7f19a413d4c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f19a41413b0 tx=0x7f199802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3740f22e34b488eb server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 10 monclient(hunting): picked mon.noname-c con 0x7f7708033550 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080cfaa0 0x7f77080c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 10 monclient(hunting): picked mon.noname-a con 0x7f77080cfaa0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77080d7c50 con 0x7f77080cfaa0 2024-04-04T21:02:27.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77080d7ad0 con 0x7f77080071c0 2024-04-04T21:02:27.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77080d7dd0 con 0x7f7708033550 2024-04-04T21:02:27.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f77153ee640 10 monclient(hunting): authenticate will time out at 1396.243404s 2024-04-04T21:02:27.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f3a093e2640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2024-04-04T21:02:27.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f3a093e2640 1 -- 172.21.15.106:0/3436398859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a040b19d0 msgr2=0x7f3a0413f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f3a093e2640 1 --2- 172.21.15.106:0/3436398859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a040b19d0 0x7f3a0413f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f3a093e2640 1 -- 172.21.15.106:0/3436398859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a040afa20 msgr2=0x7f3a04149e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f3a093e2640 1 --2- 172.21.15.106:0/3436398859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a040afa20 0x7f3a04149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f3a093e2640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f3a093e2640 1 monclient: found mon.noname-c 2024-04-04T21:02:27.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f3a093e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f3a093e2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.976+0000 7f3a093e2640 1 -- 172.21.15.106:0/3436398859 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a04144270 con 0x7f3a04143de0 2024-04-04T21:02:27.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.979+0000 7f62dffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62e0143000 0x7f62e0072b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.979+0000 7f3a093e2640 1 --2- 172.21.15.106:0/3436398859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a04143de0 0x7f3a04143280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f39ec02b820 tx=0x7f39ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.979+0000 7f3a017fa640 10 client.?.objecter ms_handle_connect 0x7f3a04143de0 2024-04-04T21:02:27.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.979+0000 7f3a017fa640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.979+0000 7f771598c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080c82b0 0x7f770815d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.979+0000 7f03fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f03fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f03fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f03fdd9b640 1 --2- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0400147150 0x7f0400120cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f771698e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f770815e1d0 0x7f7708160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f771598c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080c82b0 0x7f770815d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34814/0 (socket says 172.21.15.106:34814) 2024-04-04T21:02:27.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f771598c640 1 -- 172.21.15.106:0/1173296054 learned_addr learned my addr 172.21.15.106:0/1173296054 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f771598c640 10 monclient(hunting): get_auth_request con 0x7f77080c82b0 auth_method 0 2024-04-04T21:02:27.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f771598c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f771598c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f771598c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f62dffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62e0143000 0x7f62e0072b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50168/0 (socket says 172.21.15.106:50168) 2024-04-04T21:02:27.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f62dffff640 1 -- 172.21.15.106:0/2912605036 learned_addr learned my addr 172.21.15.106:0/2912605036 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f62dffff640 10 monclient(hunting): get_auth_request con 0x7f62e0143000 auth_method 0 2024-04-04T21:02:27.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f62dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f62dffff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7f62dffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae8bfff640 1 -- 172.21.15.106:0/3980829455 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae940b2070 con 0x7fae8c008b70 2024-04-04T21:02:27.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae8bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:27.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae8bfff640 10 monclient: dump: 2024-04-04T21:02:27.062 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.062 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.062 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae8bfff640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae8bfff640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae8bfff640 1 -- 172.21.15.106:0/3980829455 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae940a6e50 con 0x7fae8c008b70 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae9a34f640 5 monclient: authenticate success, global_id 4277 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae9a34f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae8bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae8bfff640 1 -- 172.21.15.106:0/3980829455 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae940aba90 con 0x7fae8c008b70 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae9a34f640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae9a34f640 4 set_mon_vals no callback set 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae8bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.980+0000 7fae8bfff640 10 monclient: dump: 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.064 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7f03fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7f03fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7f03fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7f03fdd9b640 10 monclient(hunting): get_auth_request con 0x7f0400147150 auth_method 0 2024-04-04T21:02:27.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7f03fdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7f03fdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7f03fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7fae9a34f640 10 monclient: shutdown 2024-04-04T21:02:27.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7fae9a34f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7fae9a34f640 1 -- 172.21.15.106:0/3980829455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae8c008b70 msgr2=0x7fae8c0cf630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7fae9a34f640 1 --2- 172.21.15.106:0/3980829455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae8c008b70 0x7fae8c0cf630 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fae94074350 tx=0x7fae940a65d0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7fae9a34f640 1 -- 172.21.15.106:0/3980829455 shutdown_connections 2024-04-04T21:02:27.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7fae9a34f640 1 --2- 172.21.15.106:0/3980829455 >> v2:172.21.15.203:3300/0 conn(0x7fae8c0cfb70 0x7fae8c0d6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7fae9a34f640 1 --2- 172.21.15.106:0/3980829455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae8c008b70 0x7fae8c0cf630 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7fae9a34f640 1 -- 172.21.15.106:0/3980829455 >> 172.21.15.106:0/3980829455 conn(0x7fae8c006250 msgr2=0x7fae8c006660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7fae9a34f640 1 -- 172.21.15.106:0/3980829455 shutdown_connections 2024-04-04T21:02:27.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.981+0000 7fae9a34f640 1 -- 172.21.15.106:0/3980829455 wait complete. 2024-04-04T21:02:27.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.982+0000 7fae9a34f640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.982+0000 7fae9a34f640 10 monclient: monmap: 2024-04-04T21:02:27.067 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.067 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.067 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.068 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.068 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.068 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.068 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.068 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.068 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.068 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.982+0000 7fae9a34f640 1 librados: starting msgr at 2024-04-04T21:02:27.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.982+0000 7fae9a34f640 1 librados: starting objecter 2024-04-04T21:02:27.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.982+0000 7fae9a34f640 1 Processor -- start 2024-04-04T21:02:27.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.982+0000 7f03fe59c640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2024-04-04T21:02:27.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.982+0000 7fae9a34f640 1 -- start start 2024-04-04T21:02:27.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.982+0000 7fae9a34f640 1 librados: setting wanted keys 2024-04-04T21:02:27.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.982+0000 7fae9a34f640 1 librados: calling monclient init 2024-04-04T21:02:27.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.982+0000 7fae9a34f640 10 monclient: init 2024-04-04T21:02:27.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7f03fe59c640 1 -- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f040014a060 msgr2=0x7f0400121210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7f03fe59c640 1 --2- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f040014a060 0x7f0400121210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7f03fe59c640 1 -- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0400147150 msgr2=0x7f0400120cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7f03fe59c640 1 --2- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0400147150 0x7f0400120cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7f03fe59c640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7f03fe59c640 1 monclient: found mon.noname-b 2024-04-04T21:02:27.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7f03fe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7f03fe59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7f03fe59c640 1 -- 172.21.15.106:0/3245008356 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04001207b0 con 0x7f04001442f0 2024-04-04T21:02:27.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7f771698e640 10 monclient(hunting): get_auth_request con 0x7f770815e1d0 auth_method 0 2024-04-04T21:02:27.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7f771698e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7f771698e640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7f771698e640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8c007400 0x7fae8c0ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 10 monclient(hunting): picked mon.noname-a con 0x7fae8c007400 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae8c008b70 0x7fae8c0c23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 10 monclient(hunting): picked mon.noname-c con 0x7fae8c008b70 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae8c0cfb70 0x7fae8c161980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 10 monclient(hunting): picked mon.noname-b con 0x7fae8c0cfb70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c0d0160 con 0x7fae8c007400 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c0cffe0 con 0x7fae8c0cfb70 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c0d6ea0 con 0x7fae8c008b70 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7fae9a34f640 10 monclient(hunting): authenticate will time out at 1396.250900s 2024-04-04T21:02:27.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.983+0000 7f771618d640 1 --2- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770802de60 0x7f77080cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.984+0000 7f771598c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.984+0000 7f771598c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.984+0000 7f62dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.984+0000 7f62dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.984+0000 7f62dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.984+0000 7f771598c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.984+0000 7f771698e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.984+0000 7f771698e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.985+0000 7f771698e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.985+0000 7f771618d640 10 monclient(hunting): get_auth_request con 0x7f770802de60 auth_method 0 2024-04-04T21:02:27.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.985+0000 7f771618d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.985+0000 7f771618d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.985+0000 7f771618d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.986+0000 7f771618d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.986+0000 7f771618d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.986+0000 7f771618d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.988+0000 7f62dffff640 10 monclient(hunting): handle_auth_done global_id 4327 payload 824 2024-04-04T21:02:27.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.988+0000 7f62dffff640 1 -- 172.21.15.106:0/2912605036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e013fd10 msgr2=0x7f62e00702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.988+0000 7f62dffff640 1 --2- 172.21.15.106:0/2912605036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e013fd10 0x7f62e00702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.988+0000 7f62dffff640 1 -- 172.21.15.106:0/2912605036 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62e006ca40 msgr2=0x7f62e00b54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.988+0000 7f62dffff640 1 --2- 172.21.15.106:0/2912605036 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62e006ca40 0x7f62e00b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.988+0000 7f62dffff640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.988+0000 7f62dffff640 1 monclient: found mon.noname-b 2024-04-04T21:02:27.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.988+0000 7f62dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.988+0000 7f62dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.988+0000 7f62dffff640 1 -- 172.21.15.106:0/2912605036 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e00720b0 con 0x7f62e0143000 2024-04-04T21:02:27.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.989+0000 7f771598c640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-04-04T21:02:27.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.989+0000 7f771598c640 1 -- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770802de60 msgr2=0x7f77080cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.989+0000 7f771598c640 1 --2- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770802de60 0x7f77080cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.989+0000 7f771598c640 1 -- 172.21.15.106:0/1173296054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f770815e1d0 msgr2=0x7f7708160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.989+0000 7f771598c640 1 --2- 172.21.15.106:0/1173296054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f770815e1d0 0x7f7708160b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.989+0000 7f771598c640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.989+0000 7f771598c640 1 monclient: found mon.noname-a 2024-04-04T21:02:27.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.989+0000 7f771598c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.989+0000 7f771598c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.989+0000 7f771598c640 1 -- 172.21.15.106:0/1173296054 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080a5f10 con 0x7f77080c82b0 2024-04-04T21:02:27.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.991+0000 7f19a159a640 1 -- 172.21.15.106:0/381212499 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998035020 con 0x7f19a4070840 2024-04-04T21:02:27.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.991+0000 7f19a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.991+0000 7f19a159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.991+0000 7f19a159a640 10 monclient: dump: 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.991+0000 7f19a159a640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.991+0000 7f19a159a640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.991+0000 7f19a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.079 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.991+0000 7f19a359e640 5 monclient: authenticate success, global_id 4302 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.991+0000 7f19a359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.991+0000 7f19a159a640 1 -- 172.21.15.106:0/381212499 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1998031640 con 0x7f19a4070840 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.991+0000 7f19a159a640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.992+0000 7f19a159a640 1 -- 172.21.15.106:0/381212499 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998035020 con 0x7f19a4070840 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.992+0000 7f19a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.992+0000 7f19a159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.992+0000 7f19a159a640 10 monclient: dump: 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.080 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.081 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.992+0000 7f19a359e640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:27.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.992+0000 7f19a359e640 4 set_mon_vals no callback set 2024-04-04T21:02:27.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.992+0000 7f19a359e640 10 monclient: shutdown 2024-04-04T21:02:27.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.992+0000 7f19a359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.992+0000 7f19a359e640 1 -- 172.21.15.106:0/381212499 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a4070840 msgr2=0x7f19a413d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.992+0000 7f19a359e640 1 --2- 172.21.15.106:0/381212499 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a4070840 0x7f19a413d4c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f19a41413b0 tx=0x7f199802b8f0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.993+0000 7f19a359e640 1 -- 172.21.15.106:0/381212499 shutdown_connections 2024-04-04T21:02:27.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.993+0000 7f19a359e640 1 --2- 172.21.15.106:0/381212499 >> v2:172.21.15.203:3300/0 conn(0x7f19a4143af0 0x7f19a4143f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.993+0000 7f19a359e640 1 --2- 172.21.15.106:0/381212499 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19a4070c20 0x7f19a413da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.993+0000 7f19a359e640 1 --2- 172.21.15.106:0/381212499 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a4070840 0x7f19a413d4c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.993+0000 7f19a359e640 1 -- 172.21.15.106:0/381212499 >> 172.21.15.106:0/381212499 conn(0x7f19a40b5e30 msgr2=0x7f19a40b6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.993+0000 7f19a359e640 1 -- 172.21.15.106:0/381212499 shutdown_connections 2024-04-04T21:02:27.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.993+0000 7f19a359e640 1 -- 172.21.15.106:0/381212499 wait complete. 2024-04-04T21:02:27.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.993+0000 7f771598c640 1 --2- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080c82b0 0x7f770815d5e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f770000d800 tx=0x7f770000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.993+0000 7f7706ffd640 10 client.?.objecter ms_handle_connect 0x7f77080c82b0 2024-04-04T21:02:27.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.993+0000 7f7706ffd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f19a359e640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f19a359e640 10 monclient: monmap: 2024-04-04T21:02:27.083 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.083 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.083 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.083 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.083 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.083 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.083 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.083 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.083 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.083 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f19a359e640 1 librados: starting msgr at 2024-04-04T21:02:27.083 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f19a359e640 1 librados: starting objecter 2024-04-04T21:02:27.084 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a017fa640 1 -- 172.21.15.106:0/3436398859 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec044070 con 0x7f3a04143de0 2024-04-04T21:02:27.084 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.084 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a017fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:27.084 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a017fa640 10 monclient: dump: 2024-04-04T21:02:27.084 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.084 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.084 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a017fa640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a017fa640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a017fa640 1 -- 172.21.15.106:0/3436398859 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ec033c00 con 0x7f3a04143de0 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a09be3640 5 monclient: authenticate success, global_id 4283 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a09be3640 10 monclient: _renew_subs 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a09be3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.085 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a09be3640 1 -- 172.21.15.106:0/3436398859 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a04144560 con 0x7f3a04143de0 2024-04-04T21:02:27.086 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f19a359e640 1 Processor -- start 2024-04-04T21:02:27.086 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a017fa640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.086 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a017fa640 1 -- 172.21.15.106:0/3436398859 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec038070 con 0x7f3a04143de0 2024-04-04T21:02:27.087 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.087 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:27.087 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a017fa640 10 monclient: dump: 2024-04-04T21:02:27.087 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.087 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.087 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.087 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.087 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.087 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.087 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.087 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.087 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.087 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.087 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a02ffd640 4 set_mon_vals no callback set 2024-04-04T21:02:27.088 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f19a359e640 1 -- start start 2024-04-04T21:02:27.088 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f19a359e640 1 librados: setting wanted keys 2024-04-04T21:02:27.088 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f19a359e640 1 librados: calling monclient init 2024-04-04T21:02:27.088 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f19a359e640 10 monclient: init 2024-04-04T21:02:27.088 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a09be3640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.088 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a09be3640 10 monclient: _renew_subs 2024-04-04T21:02:27.088 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a09be3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.088 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a09be3640 1 -- 172.21.15.106:0/3436398859 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a041437c0 con 0x7f3a04143de0 2024-04-04T21:02:27.089 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f03fe59c640 1 --2- 172.21.15.106:0/3245008356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04001442f0 0x7f04001df800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f03f400c610 tx=0x7f03f400cad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.089 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f3a09be3640 1 librados: init done 2024-04-04T21:02:27.089 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f03eeffd640 10 client.?.objecter ms_handle_connect 0x7f04001442f0 2024-04-04T21:02:27.089 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.994+0000 7f03eeffd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f7714bed640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080cfaa0 0x7f77080c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.090 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19a4070840 0x7f19a4142d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 10 monclient(hunting): picked mon.noname-c con 0x7f19a4070840 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19a4070c20 0x7f19a4139e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 10 monclient(hunting): picked mon.noname-b con 0x7f19a4070c20 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a4143af0 0x7f19a4132d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 10 monclient(hunting): picked mon.noname-a con 0x7f19a4143af0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a4144620 con 0x7f19a4143af0 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a41444a0 con 0x7f19a4070c20 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a4140310 con 0x7f19a4070840 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f19a359e640 10 monclient(hunting): authenticate will time out at 1396.262927s 2024-04-04T21:02:27.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f62dffff640 1 --2- 172.21.15.106:0/2912605036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62e0143000 0x7f62e0072b90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f62d8007e40 tx=0x7f62d800e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f62c7fff640 10 client.?.objecter ms_handle_connect 0x7f62e0143000 2024-04-04T21:02:27.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f62c7fff640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.995+0000 7f3a09be3640 10 client.4283.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.996+0000 7f3a09be3640 15 client.4283.objecter _session_command_op_assign -1 1 2024-04-04T21:02:27.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.996+0000 7f3a09be3640 20 client.4283.objecter _get_session osd=-1 returning homeless 2024-04-04T21:02:27.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.996+0000 7f3a09be3640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.996+0000 7f3a09be3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.996+0000 7f3a09be3640 1 -- 172.21.15.106:0/3436398859 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a0406a610 con 0x7f3a04143de0 2024-04-04T21:02:27.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.996+0000 7f7714bed640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080cfaa0 0x7f77080c9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34844/0 (socket says 172.21.15.106:34844) 2024-04-04T21:02:27.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.996+0000 7f7714bed640 1 -- 172.21.15.106:0/4096817127 learned_addr learned my addr 172.21.15.106:0/4096817127 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.996+0000 7f7714bed640 10 monclient(hunting): get_auth_request con 0x7f77080cfaa0 auth_method 0 2024-04-04T21:02:27.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.996+0000 7f7714bed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.996+0000 7f7714bed640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.996+0000 7f7714bed640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.997+0000 7f0d4ad9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0d4c1464e0 0x7f0d4c0b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.094 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.999+0000 7f0d4a59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d4c06e5a0 0x7f0d4c145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.095 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.999+0000 7f7714bed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.095 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.999+0000 7f7714bed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.095 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.999+0000 7f7714bed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.095 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.999+0000 7f0d4a59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d4c06e5a0 0x7f0d4c145fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50180/0 (socket says 172.21.15.106:50180) 2024-04-04T21:02:27.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.999+0000 7f0d4a59c640 1 -- 172.21.15.106:0/3580589784 learned_addr learned my addr 172.21.15.106:0/3580589784 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.999+0000 7f0d4a59c640 10 monclient(hunting): get_auth_request con 0x7f0d4c06e5a0 auth_method 0 2024-04-04T21:02:27.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.999+0000 7f0d4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.999+0000 7f0d4a59c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:26.999+0000 7f0d4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.000+0000 7f0d4ad9d640 10 monclient(hunting): get_auth_request con 0x7f0d4c1464e0 auth_method 0 2024-04-04T21:02:27.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.000+0000 7f0d4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.000+0000 7f0d4ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.000+0000 7f0d4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.000+0000 7f0d4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.000+0000 7f0d4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.000+0000 7f0d4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.097 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.000+0000 7fae98b4c640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae8c008b70 0x7fae8c0c23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.097 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.003+0000 7f7714bed640 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2024-04-04T21:02:27.097 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.003+0000 7f7714bed640 1 -- 172.21.15.106:0/4096817127 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7708033550 msgr2=0x7f770802fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.097 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.003+0000 7f7714bed640 1 --2- 172.21.15.106:0/4096817127 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7708033550 0x7f770802fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.097 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.003+0000 7f7714bed640 1 -- 172.21.15.106:0/4096817127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77080071c0 msgr2=0x7f77080c2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.098 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.003+0000 7f7714bed640 1 --2- 172.21.15.106:0/4096817127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77080071c0 0x7f77080c2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.098 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.003+0000 7f7714bed640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.098 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.003+0000 7f7714bed640 1 monclient: found mon.noname-a 2024-04-04T21:02:27.098 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.003+0000 7f7714bed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.098 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.003+0000 7f7714bed640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.098 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.003+0000 7f7714bed640 1 -- 172.21.15.106:0/4096817127 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770802df80 con 0x7f77080cfaa0 2024-04-04T21:02:27.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.004+0000 7f0d4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.004+0000 7f0d4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.004+0000 7f0d4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.100 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.005+0000 7f0d4a59c640 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2024-04-04T21:02:27.100 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.005+0000 7f0d4a59c640 1 -- 172.21.15.106:0/3580589784 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0d4c14a8b0 msgr2=0x7f0d4c0b2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.100 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.005+0000 7f0d4a59c640 1 --2- 172.21.15.106:0/3580589784 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0d4c14a8b0 0x7f0d4c0b2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.100 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.005+0000 7f0d4a59c640 1 -- 172.21.15.106:0/3580589784 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0d4c1464e0 msgr2=0x7f0d4c0b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.100 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.005+0000 7f0d4a59c640 1 --2- 172.21.15.106:0/3580589784 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0d4c1464e0 0x7f0d4c0b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.100 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.005+0000 7f0d4a59c640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.100 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.005+0000 7f0d4a59c640 1 monclient: found mon.noname-b 2024-04-04T21:02:27.100 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.005+0000 7f0d4a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.100 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.005+0000 7f0d4a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.100 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.005+0000 7f0d4a59c640 1 -- 172.21.15.106:0/3580589784 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c145530 con 0x7f0d4c06e5a0 2024-04-04T21:02:27.101 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f03eeffd640 1 -- 172.21.15.106:0/3245008356 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f4015070 con 0x7f04001442f0 2024-04-04T21:02:27.102 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f03eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.102 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f03eeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:27.102 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f03eeffd640 10 monclient: dump: 2024-04-04T21:02:27.102 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.102 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.102 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.102 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.102 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.102 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.102 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.102 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.102 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.102 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.102 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f03eeffd640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f03eeffd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f03eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f03eeffd640 1 -- 172.21.15.106:0/3245008356 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f4004220 con 0x7f04001442f0 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f0404b1c640 5 monclient: authenticate success, global_id 4324 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f0404b1c640 10 monclient: _renew_subs 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f0404b1c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f0404b1c640 1 -- 172.21.15.106:0/3245008356 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04001200b0 con 0x7f04001442f0 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f03eeffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f03eeffd640 1 -- 172.21.15.106:0/3245008356 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f4003610 con 0x7f04001442f0 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f03eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f03eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f03eeffd640 10 monclient: dump: 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f03fcd99640 4 set_mon_vals no callback set 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f0404b1c640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f0404b1c640 10 monclient: _renew_subs 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f0404b1c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f0404b1c640 1 -- 172.21.15.106:0/3245008356 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f040006fac0 con 0x7f04001442f0 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.009+0000 7f0404b1c640 1 librados: init done 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.010+0000 7f0404b1c640 10 client.4324.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.104 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.010+0000 7f0404b1c640 15 client.4324.objecter _session_command_op_assign -1 1 2024-04-04T21:02:27.105 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.010+0000 7f0404b1c640 20 client.4324.objecter _get_session osd=-1 returning homeless 2024-04-04T21:02:27.105 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.010+0000 7f0404b1c640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.105 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.010+0000 7f0404b1c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.105 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.010+0000 7f0404b1c640 1 -- 172.21.15.106:0/3245008356 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f04001205b0 con 0x7f04001442f0 2024-04-04T21:02:27.105 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.010+0000 7fae9934d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8c007400 0x7fae8c0ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.105 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.011+0000 7fae9934d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8c007400 0x7fae8c0ca1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34860/0 (socket says 172.21.15.106:34860) 2024-04-04T21:02:27.105 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.011+0000 7fae9934d640 1 -- 172.21.15.106:0/843317487 learned_addr learned my addr 172.21.15.106:0/843317487 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.105 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.011+0000 7fae9934d640 10 monclient(hunting): get_auth_request con 0x7fae8c007400 auth_method 0 2024-04-04T21:02:27.105 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.011+0000 7fae9934d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.105 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.011+0000 7fae9934d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.106 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.011+0000 7fae9934d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.106 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.012+0000 7f7714bed640 1 --2- 172.21.15.106:0/4096817127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080cfaa0 0x7f77080c9e80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f771006a920 tx=0x7f771006a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.106 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.012+0000 7f770cff9640 10 client.?.objecter ms_handle_connect 0x7f77080cfaa0 2024-04-04T21:02:27.106 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.012+0000 7f770cff9640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.106 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.014+0000 7fae9934d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.106 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.014+0000 7fae9934d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.107 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.014+0000 7fae9934d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.107 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.015+0000 7f0d4a59c640 1 --2- 172.21.15.106:0/3580589784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d4c06e5a0 0x7f0d4c145fa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00c450 tx=0x7f0d3c00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.107 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.015+0000 7f0d3affd640 10 client.?.objecter ms_handle_connect 0x7f0d4c06e5a0 2024-04-04T21:02:27.107 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.015+0000 7f0d3affd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.108 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.019+0000 7fae9934d640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-04-04T21:02:27.108 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.019+0000 7fae9934d640 1 -- 172.21.15.106:0/843317487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae8c008b70 msgr2=0x7fae8c0c23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.108 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.019+0000 7fae9934d640 1 --2- 172.21.15.106:0/843317487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae8c008b70 0x7fae8c0c23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.108 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.019+0000 7fae9934d640 1 -- 172.21.15.106:0/843317487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae8c0cfb70 msgr2=0x7fae8c161980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.108 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.019+0000 7fae9934d640 1 --2- 172.21.15.106:0/843317487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae8c0cfb70 0x7fae8c161980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.108 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.019+0000 7fae9934d640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.108 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.019+0000 7fae9934d640 1 monclient: found mon.noname-a 2024-04-04T21:02:27.108 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.019+0000 7fae9934d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.108 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.019+0000 7fae9934d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.108 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.019+0000 7fae9934d640 1 -- 172.21.15.106:0/843317487 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c0cb7e0 con 0x7fae8c007400 2024-04-04T21:02:27.110 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f7706ffd640 1 -- 172.21.15.106:0/1173296054 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770000b6c0 con 0x7f77080c82b0 2024-04-04T21:02:27.110 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f7706ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.110 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f7706ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:27.110 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f7706ffd640 10 monclient: dump: 2024-04-04T21:02:27.110 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.110 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.110 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.110 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.110 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.110 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.110 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.110 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f7706ffd640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f7706ffd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f7706ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f7706ffd640 1 -- 172.21.15.106:0/1173296054 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7700019070 con 0x7f77080c82b0 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f7706ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f7706ffd640 1 -- 172.21.15.106:0/1173296054 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770000ba80 con 0x7f77080c82b0 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f7706ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f7706ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f7706ffd640 10 monclient: dump: 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.111 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f77183f1640 5 monclient: authenticate success, global_id 4311 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f77183f1640 10 monclient: _renew_subs 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f77183f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f77183f1640 1 -- 172.21.15.106:0/1173296054 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77080a5040 con 0x7f77080c82b0 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f77183f1640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f77183f1640 10 monclient: _renew_subs 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f77183f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f77183f1640 1 -- 172.21.15.106:0/1173296054 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7708161220 con 0x7f77080c82b0 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.022+0000 7f77183f1640 1 librados: init done 2024-04-04T21:02:27.112 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.023+0000 7f771498a640 4 set_mon_vals no callback set 2024-04-04T21:02:27.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.023+0000 7f77183f1640 10 client.4311.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.023+0000 7f77183f1640 15 client.4311.objecter _session_command_op_assign -1 1 2024-04-04T21:02:27.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.023+0000 7f77183f1640 20 client.4311.objecter _get_session osd=-1 returning homeless 2024-04-04T21:02:27.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.023+0000 7f77183f1640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.023+0000 7f77183f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.023+0000 7f77183f1640 1 -- 172.21.15.106:0/1173296054 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f76f8002910 con 0x7f77080c82b0 2024-04-04T21:02:27.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f770cff9640 1 -- 172.21.15.106:0/4096817127 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100731c0 con 0x7f77080cfaa0 2024-04-04T21:02:27.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f770cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f770cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:27.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f770cff9640 10 monclient: dump: 2024-04-04T21:02:27.114 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.114 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.114 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.114 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.114 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.114 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.114 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.114 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.114 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.114 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f770cff9640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f770cff9640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f770cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f77153ee640 5 monclient: authenticate success, global_id 4314 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f77153ee640 10 monclient: _renew_subs 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f77153ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f77153ee640 1 -- 172.21.15.106:0/4096817127 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f770802e260 con 0x7f77080cfaa0 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f77153ee640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f77153ee640 10 monclient: _renew_subs 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f77153ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f77153ee640 1 -- 172.21.15.106:0/4096817127 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f770802dbc0 con 0x7f77080cfaa0 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.024+0000 7f77153ee640 1 librados: init done 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.025+0000 7f77153ee640 10 client.4314.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.025+0000 7f77153ee640 15 client.4314.objecter _session_command_op_assign -1 1 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.025+0000 7f77153ee640 20 client.4314.objecter _get_session osd=-1 returning homeless 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.025+0000 7f77153ee640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.025+0000 7f77153ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.116 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.025+0000 7f77153ee640 1 -- 172.21.15.106:0/4096817127 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f77080057d0 con 0x7f77080cfaa0 2024-04-04T21:02:27.116 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.026+0000 7fae9934d640 1 --2- 172.21.15.106:0/843317487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8c007400 0x7fae8c0ca1a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fae9000c460 tx=0x7fae9000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.116 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.027+0000 7fae89ffb640 10 client.?.objecter ms_handle_connect 0x7fae8c007400 2024-04-04T21:02:27.116 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.027+0000 7fae89ffb640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.117 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.028+0000 7f3a017fa640 1 -- 172.21.15.106:0/3436398859 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f39ec047070 con 0x7f3a04143de0 2024-04-04T21:02:27.117 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.028+0000 7f3a017fa640 1 --2- 172.21.15.106:0/3436398859 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f39e406ceb0 0x7f39e406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.117 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.029+0000 7f3a08be1640 1 --2- 172.21.15.106:0/3436398859 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f39e406ceb0 0x7f39e406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.117 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.029+0000 7f3a08be1640 10 monclient: get_auth_request con 0x7f39e406ceb0 auth_method 0 2024-04-04T21:02:27.117 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.029+0000 7f3a017fa640 10 client.4283.objecter ms_handle_connect 0x7f39e406ceb0 2024-04-04T21:02:27.117 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.029+0000 7f3a08be1640 1 --2- 172.21.15.106:0/3436398859 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f39e406ceb0 0x7f39e406f370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f39f4001560 tx=0x7f39f4007990 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.031+0000 7f3a017fa640 1 -- 172.21.15.106:0/3436398859 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f39ec078e10 con 0x7f3a04143de0 2024-04-04T21:02:27.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.031+0000 7f3a017fa640 10 client.4283.objecter ms_dispatch 0x7f3a04000b90 osd_map(26..26 src has 1..26) v4 2024-04-04T21:02:27.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.031+0000 7f3a017fa640 3 client.4283.objecter handle_osd_map got epochs [26,26] > 0 2024-04-04T21:02:27.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.031+0000 7f3a017fa640 3 client.4283.objecter handle_osd_map decoding full epoch 26 2024-04-04T21:02:27.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.031+0000 7f3a017fa640 10 client.4283.objecter checking command 1 2024-04-04T21:02:27.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.031+0000 7f3a017fa640 1 --2- 172.21.15.106:0/3436398859 >> [v2:172.21.15.203:6800/696438187,v1:172.21.15.203:6801/696438187] conn(0x7f39e40732f0 0x7f39e4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.032+0000 7f3a017fa640 20 client.4283.objecter _get_session s=0x7f39e4072c20 osd=5 3 2024-04-04T21:02:27.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.032+0000 7f3a017fa640 20 client.4283.objecter put_session s=0x7f39e4072c20 osd=5 3 2024-04-04T21:02:27.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.032+0000 7f3a017fa640 15 client.4283.objecter _session_command_op_remove -1 1 2024-04-04T21:02:27.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.032+0000 7f3a017fa640 20 client.4283.objecter _get_session s=0x7f39e4072c20 osd=5 3 2024-04-04T21:02:27.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.032+0000 7f3a017fa640 20 client.4283.objecter get_session s=0x7f39e4072c20 osd=5 3 2024-04-04T21:02:27.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.032+0000 7f3a017fa640 15 client.4283.objecter _session_command_op_assign 5 1 2024-04-04T21:02:27.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.032+0000 7f3a017fa640 20 client.4283.objecter put_session s=0x7f39e4072c20 osd=5 4 2024-04-04T21:02:27.118 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.032+0000 7f3a017fa640 10 client.4283.objecter _send_command 1 2024-04-04T21:02:27.119 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.032+0000 7f3a017fa640 1 -- 172.21.15.106:0/3436398859 --> [v2:172.21.15.203:6800/696438187,v1:172.21.15.203:6801/696438187] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39e4075dc0 con 0x7f39e40732f0 2024-04-04T21:02:27.119 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.032+0000 7f3a017fa640 20 client.4283.objecter dump_active .. 0 homeless 2024-04-04T21:02:27.119 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.032+0000 7f03eeffd640 1 -- 172.21.15.106:0/3245008356 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f03f40037b0 con 0x7f04001442f0 2024-04-04T21:02:27.119 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.032+0000 7f3a03fff640 1 --2- 172.21.15.106:0/3436398859 >> [v2:172.21.15.203:6800/696438187,v1:172.21.15.203:6801/696438187] conn(0x7f39e40732f0 0x7f39e4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.119 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.032+0000 7f03eeffd640 1 --2- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f03e006ceb0 0x7f03e006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.033+0000 7f3a03fff640 10 monclient: get_auth_request con 0x7f39e40732f0 auth_method 0 2024-04-04T21:02:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.033+0000 7f03fdd9b640 1 --2- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f03e006ceb0 0x7f03e006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.033+0000 7f03fdd9b640 10 monclient: get_auth_request con 0x7f03e006ceb0 auth_method 0 2024-04-04T21:02:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.033+0000 7f03eeffd640 10 client.4324.objecter ms_handle_connect 0x7f03e006ceb0 2024-04-04T21:02:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.033+0000 7f03fdd9b640 1 --2- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f03e006ceb0 0x7f03e006f370 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f03f8035000 tx=0x7f03f8030000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.036+0000 7f3a017fa640 10 client.4283.objecter ms_handle_connect 0x7f39e40732f0 2024-04-04T21:02:27.121 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.036+0000 7f3a03fff640 1 --2- 172.21.15.106:0/3436398859 >> [v2:172.21.15.203:6800/696438187,v1:172.21.15.203:6801/696438187] conn(0x7f39e40732f0 0x7f39e4075710 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.037+0000 7f19a259c640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19a4070840 0x7f19a4142d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.038+0000 7f19a259c640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19a4070840 0x7f19a4142d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:32810/0 (socket says 172.21.15.106:32810) 2024-04-04T21:02:27.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.038+0000 7f19a259c640 1 -- 172.21.15.106:0/1332423717 learned_addr learned my addr 172.21.15.106:0/1332423717 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.038+0000 7f19a259c640 10 monclient(hunting): get_auth_request con 0x7f19a4070840 auth_method 0 2024-04-04T21:02:27.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.038+0000 7f19a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.038+0000 7f19a259c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.038+0000 7f19a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.039+0000 7f19a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.039+0000 7f19a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.039+0000 7f19a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.122 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f19a259c640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2024-04-04T21:02:27.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 1 -- 172.21.15.106:0/3245008356 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f03f4056df0 con 0x7f04001442f0 2024-04-04T21:02:27.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 10 client.4324.objecter ms_dispatch 0x7f0400000b90 osd_map(26..26 src has 1..26) v4 2024-04-04T21:02:27.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 3 client.4324.objecter handle_osd_map got epochs [26,26] > 0 2024-04-04T21:02:27.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 3 client.4324.objecter handle_osd_map decoding full epoch 26 2024-04-04T21:02:27.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f19a259c640 1 -- 172.21.15.106:0/1332423717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19a4070c20 msgr2=0x7f19a4139e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.123 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f19a1d9b640 1 --2- 172.21.15.106:0/1332423717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19a4070c20 0x7f19a4139e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 10 client.4324.objecter checking command 1 2024-04-04T21:02:27.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 1 --2- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:6818/2744620362,v1:172.21.15.106:6819/2744620362] conn(0x7f03e00732f0 0x7f03e0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f19a259c640 1 --2- 172.21.15.106:0/1332423717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19a4070c20 0x7f19a4139e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 20 client.4324.objecter _get_session s=0x7f03e0072c20 osd=0 3 2024-04-04T21:02:27.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 20 client.4324.objecter put_session s=0x7f03e0072c20 osd=0 3 2024-04-04T21:02:27.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 15 client.4324.objecter _session_command_op_remove -1 1 2024-04-04T21:02:27.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 20 client.4324.objecter _get_session s=0x7f03e0072c20 osd=0 3 2024-04-04T21:02:27.124 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 20 client.4324.objecter get_session s=0x7f03e0072c20 osd=0 3 2024-04-04T21:02:27.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f19a259c640 1 -- 172.21.15.106:0/1332423717 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a4143af0 msgr2=0x7f19a4132d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 15 client.4324.objecter _session_command_op_assign 0 1 2024-04-04T21:02:27.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 20 client.4324.objecter put_session s=0x7f03e0072c20 osd=0 4 2024-04-04T21:02:27.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 10 client.4324.objecter _send_command 1 2024-04-04T21:02:27.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 1 -- 172.21.15.106:0/3245008356 --> [v2:172.21.15.106:6818/2744620362,v1:172.21.15.106:6819/2744620362] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f03f4008e50 con 0x7f03e00732f0 2024-04-04T21:02:27.125 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f03eeffd640 20 client.4324.objecter dump_active .. 0 homeless 2024-04-04T21:02:27.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f19a259c640 1 --2- 172.21.15.106:0/1332423717 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19a4143af0 0x7f19a4132d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f19a259c640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f19a259c640 1 monclient: found mon.noname-c 2024-04-04T21:02:27.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f19a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f19a259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f19a259c640 1 -- 172.21.15.106:0/1332423717 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a4139800 con 0x7f19a4070840 2024-04-04T21:02:27.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f62c7fff640 1 -- 172.21.15.106:0/2912605036 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d8014620 con 0x7f62e0143000 2024-04-04T21:02:27.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f62c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f62c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:27.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f62c7fff640 10 monclient: dump: 2024-04-04T21:02:27.127 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.127 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.127 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.127 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.128 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.128 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.128 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.128 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.128 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.128 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.042+0000 7f62c7fff640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62c7fff640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62c7fff640 1 -- 172.21.15.106:0/2912605036 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d8019070 con 0x7f62e0143000 2024-04-04T21:02:27.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62c7fff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f03fed9d640 1 --2- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:6818/2744620362,v1:172.21.15.106:6819/2744620362] conn(0x7f03e00732f0 0x7f03e0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f03fed9d640 10 monclient: get_auth_request con 0x7f03e00732f0 auth_method 0 2024-04-04T21:02:27.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62ddffb640 4 set_mon_vals no callback set 2024-04-04T21:02:27.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62e5da4640 5 monclient: authenticate success, global_id 4327 2024-04-04T21:02:27.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62c7fff640 1 -- 172.21.15.106:0/2912605036 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d8014a20 con 0x7f62e0143000 2024-04-04T21:02:27.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62e5da4640 10 monclient: _renew_subs 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62e5da4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62e5da4640 1 -- 172.21.15.106:0/2912605036 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e0072670 con 0x7f62e0143000 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62c7fff640 10 monclient: dump: 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.130 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62e5da4640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62e5da4640 10 monclient: _renew_subs 2024-04-04T21:02:27.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62e5da4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62e5da4640 1 -- 172.21.15.106:0/2912605036 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e0144e00 con 0x7f62e0143000 2024-04-04T21:02:27.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.043+0000 7f62e5da4640 1 librados: init done 2024-04-04T21:02:27.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.044+0000 7f62e5da4640 10 client.4327.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.044+0000 7f62e5da4640 15 client.4327.objecter _session_command_op_assign -1 1 2024-04-04T21:02:27.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.044+0000 7f62e5da4640 20 client.4327.objecter _get_session osd=-1 returning homeless 2024-04-04T21:02:27.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.044+0000 7f62e5da4640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.044+0000 7f62e5da4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.044+0000 7f62e5da4640 1 -- 172.21.15.106:0/2912605036 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f62e004f050 con 0x7f62e0143000 2024-04-04T21:02:27.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.045+0000 7f19a259c640 1 --2- 172.21.15.106:0/1332423717 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19a4070840 0x7f19a4142d10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1998000c00 tx=0x7f19980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.045+0000 7f1992ffd640 10 client.?.objecter ms_handle_connect 0x7f19a4070840 2024-04-04T21:02:27.132 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.045+0000 7f1992ffd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.046+0000 7f0d3affd640 1 -- 172.21.15.106:0/3580589784 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c017070 con 0x7f0d4c06e5a0 2024-04-04T21:02:27.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.046+0000 7f0d3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.046+0000 7f0d3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:27.133 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.046+0000 7f0d3affd640 10 monclient: dump: 2024-04-04T21:02:27.133 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.133 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.133 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.133 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.133 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.133 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.133 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.133 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.133 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.046+0000 7f0d3affd640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.046+0000 7f0d3affd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.046+0000 7f0d3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.046+0000 7f0d3affd640 1 -- 172.21.15.106:0/3580589784 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3c005ce0 con 0x7f0d4c06e5a0 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.046+0000 7f0d3affd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.046+0000 7f0d3affd640 1 -- 172.21.15.106:0/3580589784 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c0072b0 con 0x7f0d4c06e5a0 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.046+0000 7f0d3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.046+0000 7f0d3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.046+0000 7f0d3affd640 10 monclient: dump: 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.134 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.046+0000 7f0d48d99640 4 set_mon_vals no callback set 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.047+0000 7f0d4b59e640 5 monclient: authenticate success, global_id 4333 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.047+0000 7f0d4b59e640 10 monclient: _renew_subs 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.047+0000 7f0d4b59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.047+0000 7f0d4b59e640 1 -- 172.21.15.106:0/3580589784 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4c145810 con 0x7f0d4c06e5a0 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.047+0000 7f0d4b59e640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.047+0000 7f0d4b59e640 10 monclient: _renew_subs 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.047+0000 7f0d4b59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.047+0000 7f0d4b59e640 1 -- 172.21.15.106:0/3580589784 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4c0b31b0 con 0x7f0d4c06e5a0 2024-04-04T21:02:27.135 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.047+0000 7f0d4b59e640 1 librados: init done 2024-04-04T21:02:27.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.048+0000 7f0d4b59e640 10 client.4333.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.048+0000 7f0d4b59e640 15 client.4333.objecter _session_command_op_assign -1 1 2024-04-04T21:02:27.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.048+0000 7f0d4b59e640 20 client.4333.objecter _get_session osd=-1 returning homeless 2024-04-04T21:02:27.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.048+0000 7f0d4b59e640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.048+0000 7f0d4b59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.048+0000 7f0d4b59e640 1 -- 172.21.15.106:0/3580589784 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0d4c146b40 con 0x7f0d4c06e5a0 2024-04-04T21:02:27.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.048+0000 7f03fed9d640 1 --2- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:6818/2744620362,v1:172.21.15.106:6819/2744620362] conn(0x7f03e00732f0 0x7f03e0075710 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.048+0000 7f03eeffd640 10 client.4324.objecter ms_handle_connect 0x7f03e00732f0 2024-04-04T21:02:27.137 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f1992ffd640 1 -- 172.21.15.106:0/1332423717 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998035020 con 0x7f19a4070840 2024-04-04T21:02:27.137 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f1992ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.137 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f1992ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:27.137 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f1992ffd640 10 monclient: dump: 2024-04-04T21:02:27.137 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.137 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f1992ffd640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f1992ffd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f1992ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f19a359e640 5 monclient: authenticate success, global_id 4292 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f19a359e640 10 monclient: _renew_subs 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f19a359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f19a359e640 1 -- 172.21.15.106:0/1332423717 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a4139a70 con 0x7f19a4070840 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f1992ffd640 1 -- 172.21.15.106:0/1332423717 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199803aa60 con 0x7f19a4070840 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f1992ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f1992ffd640 1 -- 172.21.15.106:0/1332423717 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998035020 con 0x7f19a4070840 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f1992ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f19a359e640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f1992ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f1992ffd640 10 monclient: dump: 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.139 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.140 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.140 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f19a359e640 10 monclient: _renew_subs 2024-04-04T21:02:27.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f19a359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f19a359e640 1 -- 172.21.15.106:0/1332423717 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a41d3090 con 0x7f19a4070840 2024-04-04T21:02:27.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f19a0d99640 4 set_mon_vals no callback set 2024-04-04T21:02:27.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.049+0000 7f19a359e640 1 librados: init done 2024-04-04T21:02:27.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.050+0000 7f19a359e640 10 client.4292.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.050+0000 7f19a359e640 15 client.4292.objecter _session_command_op_assign -1 1 2024-04-04T21:02:27.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.050+0000 7f19a359e640 20 client.4292.objecter _get_session osd=-1 returning homeless 2024-04-04T21:02:27.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.050+0000 7f19a359e640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.050+0000 7f19a359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.050+0000 7f19a359e640 1 -- 172.21.15.106:0/1332423717 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f19a406a610 con 0x7f19a4070840 2024-04-04T21:02:27.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.051+0000 7f7706ffd640 1 -- 172.21.15.106:0/1173296054 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f770000bc20 con 0x7f77080c82b0 2024-04-04T21:02:27.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.052+0000 7f7706ffd640 1 --2- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f76f406ceb0 0x7f76f406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.052+0000 7f771618d640 1 --2- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f76f406ceb0 0x7f76f406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.052+0000 7f771618d640 10 monclient: get_auth_request con 0x7f76f406ceb0 auth_method 0 2024-04-04T21:02:27.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.053+0000 7f771618d640 1 --2- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f76f406ceb0 0x7f76f406f370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7708033d70 tx=0x7f77100aa000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.053+0000 7f7706ffd640 10 client.4311.objecter ms_handle_connect 0x7f76f406ceb0 2024-04-04T21:02:27.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.060+0000 7f7706ffd640 1 -- 172.21.15.106:0/1173296054 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7700057810 con 0x7f77080c82b0 2024-04-04T21:02:27.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 10 client.4311.objecter ms_dispatch 0x7f7710000b90 osd_map(26..26 src has 1..26) v4 2024-04-04T21:02:27.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 3 client.4311.objecter handle_osd_map got epochs [26,26] > 0 2024-04-04T21:02:27.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 3 client.4311.objecter handle_osd_map decoding full epoch 26 2024-04-04T21:02:27.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 10 client.4311.objecter checking command 1 2024-04-04T21:02:27.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 1 --2- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:6826/2321665741,v1:172.21.15.106:6827/2321665741] conn(0x7f76f40732f0 0x7f76f4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 20 client.4311.objecter _get_session s=0x7f76f4072c20 osd=1 3 2024-04-04T21:02:27.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 20 client.4311.objecter put_session s=0x7f76f4072c20 osd=1 3 2024-04-04T21:02:27.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 15 client.4311.objecter _session_command_op_remove -1 1 2024-04-04T21:02:27.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 20 client.4311.objecter _get_session s=0x7f76f4072c20 osd=1 3 2024-04-04T21:02:27.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 20 client.4311.objecter get_session s=0x7f76f4072c20 osd=1 3 2024-04-04T21:02:27.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 15 client.4311.objecter _session_command_op_assign 1 1 2024-04-04T21:02:27.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 20 client.4311.objecter put_session s=0x7f76f4072c20 osd=1 4 2024-04-04T21:02:27.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 10 client.4311.objecter _send_command 1 2024-04-04T21:02:27.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 1 -- 172.21.15.106:0/1173296054 --> [v2:172.21.15.106:6826/2321665741,v1:172.21.15.106:6827/2321665741] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7700016e70 con 0x7f76f40732f0 2024-04-04T21:02:27.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 20 client.4311.objecter dump_active .. 0 homeless 2024-04-04T21:02:27.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 1 -- 172.21.15.106:0/1173296054 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7700055a10 con 0x7f77080c82b0 2024-04-04T21:02:27.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.061+0000 7f7706ffd640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-04-04T21:02:27.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.062+0000 7f771698e640 1 --2- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:6826/2321665741,v1:172.21.15.106:6827/2321665741] conn(0x7f76f40732f0 0x7f76f4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.062+0000 7f770cff9640 1 -- 172.21.15.106:0/4096817127 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77080057d0 con 0x7f77080cfaa0 2024-04-04T21:02:27.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.062+0000 7f770cff9640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.062+0000 7f770e7fc640 4 set_mon_vals no callback set 2024-04-04T21:02:27.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.063+0000 7f771698e640 10 monclient: get_auth_request con 0x7f76f40732f0 auth_method 0 2024-04-04T21:02:27.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.063+0000 7f770cff9640 1 -- 172.21.15.106:0/4096817127 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710098060 con 0x7f77080cfaa0 2024-04-04T21:02:27.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.063+0000 7f770cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.063+0000 7f770cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:27.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.063+0000 7f770cff9640 10 monclient: dump: 2024-04-04T21:02:27.146 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.146 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.146 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.146 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.146 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.146 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.146 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.146 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.146 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.146 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.066+0000 7f771698e640 1 --2- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:6826/2321665741,v1:172.21.15.106:6827/2321665741] conn(0x7f76f40732f0 0x7f76f4075710 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.066+0000 7f7706ffd640 10 client.4311.objecter ms_handle_connect 0x7f76f40732f0 2024-04-04T21:02:27.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.066+0000 7f62c7fff640 1 -- 172.21.15.106:0/2912605036 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f62d8014bc0 con 0x7f62e0143000 2024-04-04T21:02:27.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.066+0000 7fae89ffb640 1 -- 172.21.15.106:0/843317487 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90011790 con 0x7fae8c007400 2024-04-04T21:02:27.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae89ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae89ffb640 10 monclient: dump: 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae89ffb640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae89ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae9a34f640 5 monclient: authenticate success, global_id 4320 2024-04-04T21:02:27.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae9a34f640 10 monclient: _renew_subs 2024-04-04T21:02:27.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae9a34f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae9a34f640 1 -- 172.21.15.106:0/843317487 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8c0cbc00 con 0x7fae8c007400 2024-04-04T21:02:27.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7f62c7fff640 1 --2- 172.21.15.106:0/2912605036 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f62cc06ce90 0x7f62cc06f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae9a34f640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae9a34f640 10 monclient: _renew_subs 2024-04-04T21:02:27.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae9a34f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae9a34f640 1 -- 172.21.15.106:0/843317487 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c0c1ca0 con 0x7fae8c007400 2024-04-04T21:02:27.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.067+0000 7fae9a34f640 1 librados: init done 2024-04-04T21:02:27.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.068+0000 7f62df7fe640 1 --2- 172.21.15.106:0/2912605036 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f62cc06ce90 0x7f62cc06f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.068+0000 7f62df7fe640 10 monclient: get_auth_request con 0x7f62cc06ce90 auth_method 0 2024-04-04T21:02:27.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.068+0000 7fae9a34f640 10 client.4320.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.068+0000 7fae9a34f640 15 client.4320.objecter _session_command_op_assign -1 1 2024-04-04T21:02:27.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.068+0000 7fae9a34f640 20 client.4320.objecter _get_session osd=-1 returning homeless 2024-04-04T21:02:27.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.068+0000 7fae9a34f640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.068+0000 7fae9a34f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.068+0000 7fae9a34f640 1 -- 172.21.15.106:0/843317487 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fae8c1617d0 con 0x7fae8c007400 2024-04-04T21:02:27.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.068+0000 7f62df7fe640 1 --2- 172.21.15.106:0/2912605036 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f62cc06ce90 0x7f62cc06f350 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f62d4036000 tx=0x7f62d4030000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.068+0000 7f62c7fff640 10 client.4327.objecter ms_handle_connect 0x7f62cc06ce90 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 1 -- 172.21.15.106:0/2912605036 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f62d8058170 con 0x7f62e0143000 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 10 client.4327.objecter ms_dispatch 0x7f62e0000b90 osd_map(26..26 src has 1..26) v4 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 3 client.4327.objecter handle_osd_map got epochs [26,26] > 0 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 3 client.4327.objecter handle_osd_map decoding full epoch 26 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 10 client.4327.objecter checking command 1 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 1 --2- 172.21.15.106:0/2912605036 >> [v2:172.21.15.106:6802/873500775,v1:172.21.15.106:6803/873500775] conn(0x7f62cc0732d0 0x7f62cc0756f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 20 client.4327.objecter _get_session s=0x7f62cc072c00 osd=3 3 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 20 client.4327.objecter put_session s=0x7f62cc072c00 osd=3 3 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 15 client.4327.objecter _session_command_op_remove -1 1 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 20 client.4327.objecter _get_session s=0x7f62cc072c00 osd=3 3 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 20 client.4327.objecter get_session s=0x7f62cc072c00 osd=3 3 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 15 client.4327.objecter _session_command_op_assign 3 1 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 20 client.4327.objecter put_session s=0x7f62cc072c00 osd=3 4 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 10 client.4327.objecter _send_command 1 2024-04-04T21:02:27.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 1 -- 172.21.15.106:0/2912605036 --> [v2:172.21.15.106:6802/873500775,v1:172.21.15.106:6803/873500775] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62cc075da0 con 0x7f62cc0732d0 2024-04-04T21:02:27.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.069+0000 7f62c7fff640 20 client.4327.objecter dump_active .. 0 homeless 2024-04-04T21:02:27.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.072+0000 7fae89ffb640 1 -- 172.21.15.106:0/843317487 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae90017070 con 0x7fae8c007400 2024-04-04T21:02:27.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.072+0000 7fae89ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.072+0000 7fae8b7fe640 4 set_mon_vals no callback set 2024-04-04T21:02:27.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.073+0000 7fae89ffb640 1 -- 172.21.15.106:0/843317487 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90011b30 con 0x7fae8c007400 2024-04-04T21:02:27.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.073+0000 7fae89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.073+0000 7fae89ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:27.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.073+0000 7fae89ffb640 10 monclient: dump: 2024-04-04T21:02:27.154 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.154 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.154 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.154 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.154 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.154 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.154 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.154 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.154 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.154 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.078+0000 7f1992ffd640 1 -- 172.21.15.106:0/1332423717 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f199804b060 con 0x7f19a4070840 2024-04-04T21:02:27.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.079+0000 7f1992ffd640 1 --2- 172.21.15.106:0/1332423717 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f198406d060 0x7f198406f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.079+0000 7f3a017fa640 1 -- 172.21.15.106:0/3436398859 <== osd.5 v2:172.21.15.203:6800/696438187 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f39e4075dc0 con 0x7f39e40732f0 2024-04-04T21:02:27.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.079+0000 7f3a017fa640 10 client.4283.objecter ms_dispatch 0x7f3a04000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:02:27.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.079+0000 7f3a017fa640 10 client.4283.objecter _finish_command 1 = system:0 2024-04-04T21:02:27.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.079+0000 7f3a017fa640 20 client.4283.objecter put_session s=0x7f39e4072c20 osd=5 4 2024-04-04T21:02:27.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.079+0000 7f3a017fa640 15 client.4283.objecter _session_command_op_remove 5 1 2024-04-04T21:02:27.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.079+0000 7f19a1d9b640 1 --2- 172.21.15.106:0/1332423717 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f198406d060 0x7f198406f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.079+0000 7f19a1d9b640 10 monclient: get_auth_request con 0x7f198406d060 auth_method 0 2024-04-04T21:02:27.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.080+0000 7f19a1d9b640 1 --2- 172.21.15.106:0/1332423717 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f198406d060 0x7f198406f520 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f19a40b52b0 tx=0x7f1994008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.080+0000 7f1992ffd640 10 client.4292.objecter ms_handle_connect 0x7f198406d060 2024-04-04T21:02:27.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 1 -- 172.21.15.106:0/1332423717 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f199804be10 con 0x7f19a4070840 2024-04-04T21:02:27.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 10 client.4292.objecter ms_dispatch 0x7f19a4000b90 osd_map(26..26 src has 1..26) v4 2024-04-04T21:02:27.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 3 client.4292.objecter handle_osd_map got epochs [26,26] > 0 2024-04-04T21:02:27.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 3 client.4292.objecter handle_osd_map decoding full epoch 26 2024-04-04T21:02:27.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 10 client.4292.objecter checking command 1 2024-04-04T21:02:27.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 1 --2- 172.21.15.106:0/1332423717 >> [v2:172.21.15.203:6816/2758852830,v1:172.21.15.203:6817/2758852830] conn(0x7f19840734a0 0x7f19840758c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 20 client.4292.objecter _get_session s=0x7f1984072dd0 osd=6 3 2024-04-04T21:02:27.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 20 client.4292.objecter put_session s=0x7f1984072dd0 osd=6 3 2024-04-04T21:02:27.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 15 client.4292.objecter _session_command_op_remove -1 1 2024-04-04T21:02:27.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 20 client.4292.objecter _get_session s=0x7f1984072dd0 osd=6 3 2024-04-04T21:02:27.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 20 client.4292.objecter get_session s=0x7f1984072dd0 osd=6 3 2024-04-04T21:02:27.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 15 client.4292.objecter _session_command_op_assign 6 1 2024-04-04T21:02:27.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 20 client.4292.objecter put_session s=0x7f1984072dd0 osd=6 4 2024-04-04T21:02:27.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 10 client.4292.objecter _send_command 1 2024-04-04T21:02:27.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 1 -- 172.21.15.106:0/1332423717 --> [v2:172.21.15.203:6816/2758852830,v1:172.21.15.203:6817/2758852830] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1998008f30 con 0x7f19840734a0 2024-04-04T21:02:27.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.083+0000 7f1992ffd640 20 client.4292.objecter dump_active .. 0 homeless 2024-04-04T21:02:27.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.084+0000 7f19a2d9d640 1 --2- 172.21.15.106:0/1332423717 >> [v2:172.21.15.203:6816/2758852830,v1:172.21.15.203:6817/2758852830] conn(0x7f19840734a0 0x7f19840758c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.085+0000 7f19a2d9d640 10 monclient: get_auth_request con 0x7f19840734a0 auth_method 0 2024-04-04T21:02:27.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.091+0000 7f0d3affd640 1 -- 172.21.15.106:0/3580589784 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0d3c01a070 con 0x7f0d4c06e5a0 2024-04-04T21:02:27.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.091+0000 7f19a2d9d640 1 --2- 172.21.15.106:0/1332423717 >> [v2:172.21.15.203:6816/2758852830,v1:172.21.15.203:6817/2758852830] conn(0x7f19840734a0 0x7f19840758c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.091+0000 7f1992ffd640 10 client.4292.objecter ms_handle_connect 0x7f19840734a0 2024-04-04T21:02:27.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.091+0000 7f0d3affd640 1 --2- 172.21.15.106:0/3580589784 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f0d2c06cee0 0x7f0d2c06f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.091+0000 7f770cff9640 1 -- 172.21.15.106:0/4096817127 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f771009c3e0 con 0x7f77080cfaa0 2024-04-04T21:02:27.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.092+0000 7f0d49d9b640 1 --2- 172.21.15.106:0/3580589784 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f0d2c06cee0 0x7f0d2c06f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.092+0000 7f770cff9640 1 --2- 172.21.15.106:0/4096817127 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f76f406ce60 0x7f76f406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.092+0000 7f770ffff640 1 --2- 172.21.15.106:0/4096817127 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f76f406ce60 0x7f76f406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.092+0000 7f770ffff640 10 monclient: get_auth_request con 0x7f76f406ce60 auth_method 0 2024-04-04T21:02:27.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.092+0000 7f0d49d9b640 10 monclient: get_auth_request con 0x7f0d2c06cee0 auth_method 0 2024-04-04T21:02:27.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.092+0000 7f0d49d9b640 1 --2- 172.21.15.106:0/3580589784 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f0d2c06cee0 0x7f0d2c06f3a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0d440012e0 tx=0x7f0d44007d30 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.092+0000 7f0d3affd640 10 client.4333.objecter ms_handle_connect 0x7f0d2c06cee0 2024-04-04T21:02:27.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.092+0000 7f770ffff640 1 --2- 172.21.15.106:0/4096817127 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f76f406ce60 0x7f76f406f320 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7704035000 tx=0x7f7704030000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f770cff9640 10 client.4314.objecter ms_handle_connect 0x7f76f406ce60 2024-04-04T21:02:27.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 1 -- 172.21.15.106:0/3580589784 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0d3c0561e0 con 0x7f0d4c06e5a0 2024-04-04T21:02:27.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 10 client.4333.objecter ms_dispatch 0x7f0d4c000b90 osd_map(26..26 src has 1..26) v4 2024-04-04T21:02:27.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 3 client.4333.objecter handle_osd_map got epochs [26,26] > 0 2024-04-04T21:02:27.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 3 client.4333.objecter handle_osd_map decoding full epoch 26 2024-04-04T21:02:27.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 10 client.4333.objecter checking command 1 2024-04-04T21:02:27.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 1 --2- 172.21.15.106:0/3580589784 >> [v2:172.21.15.106:6810/459340769,v1:172.21.15.106:6811/459340769] conn(0x7f0d2c073320 0x7f0d2c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 20 client.4333.objecter _get_session s=0x7f0d2c072c50 osd=2 3 2024-04-04T21:02:27.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 20 client.4333.objecter put_session s=0x7f0d2c072c50 osd=2 3 2024-04-04T21:02:27.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 15 client.4333.objecter _session_command_op_remove -1 1 2024-04-04T21:02:27.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 20 client.4333.objecter _get_session s=0x7f0d2c072c50 osd=2 3 2024-04-04T21:02:27.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 20 client.4333.objecter get_session s=0x7f0d2c072c50 osd=2 3 2024-04-04T21:02:27.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 15 client.4333.objecter _session_command_op_assign 2 1 2024-04-04T21:02:27.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 20 client.4333.objecter put_session s=0x7f0d2c072c50 osd=2 4 2024-04-04T21:02:27.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 10 client.4333.objecter _send_command 1 2024-04-04T21:02:27.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 1 -- 172.21.15.106:0/3580589784 --> [v2:172.21.15.106:6810/459340769,v1:172.21.15.106:6811/459340769] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d3c006e50 con 0x7f0d2c073320 2024-04-04T21:02:27.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.093+0000 7f0d3affd640 20 client.4333.objecter dump_active .. 0 homeless 2024-04-04T21:02:27.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.094+0000 7f0d4ad9d640 1 --2- 172.21.15.106:0/3580589784 >> [v2:172.21.15.106:6810/459340769,v1:172.21.15.106:6811/459340769] conn(0x7f0d2c073320 0x7f0d2c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.094+0000 7f0d4ad9d640 10 monclient: get_auth_request con 0x7f0d2c073320 auth_method 0 2024-04-04T21:02:27.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.099+0000 7f62deffd640 1 --2- 172.21.15.106:0/2912605036 >> [v2:172.21.15.106:6802/873500775,v1:172.21.15.106:6803/873500775] conn(0x7f62cc0732d0 0x7f62cc0756f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.100+0000 7f62deffd640 10 monclient: get_auth_request con 0x7f62cc0732d0 auth_method 0 2024-04-04T21:02:27.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.100+0000 7f0d4ad9d640 1 --2- 172.21.15.106:0/3580589784 >> [v2:172.21.15.106:6810/459340769,v1:172.21.15.106:6811/459340769] conn(0x7f0d2c073320 0x7f0d2c075740 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.100+0000 7f0d3affd640 10 client.4333.objecter ms_handle_connect 0x7f0d2c073320 2024-04-04T21:02:27.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.107+0000 7f62deffd640 1 --2- 172.21.15.106:0/2912605036 >> [v2:172.21.15.106:6802/873500775,v1:172.21.15.106:6803/873500775] conn(0x7f62cc0732d0 0x7f62cc0756f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.107+0000 7f62c7fff640 10 client.4327.objecter ms_handle_connect 0x7f62cc0732d0 2024-04-04T21:02:27.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.108+0000 7f3a09be3640 10 client.4283.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-04T21:02:27.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.108+0000 7f3a09be3640 15 client.4283.objecter _session_command_op_assign -1 2 2024-04-04T21:02:27.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.108+0000 7f3a09be3640 20 client.4283.objecter _get_session s=0x7f39e4072c20 osd=5 3 2024-04-04T21:02:27.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.108+0000 7f3a09be3640 20 client.4283.objecter put_session s=0x7f39e4072c20 osd=5 3 2024-04-04T21:02:27.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.108+0000 7f3a09be3640 20 client.4283.objecter _get_session s=0x7f39e4072c20 osd=5 3 2024-04-04T21:02:27.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.108+0000 7f3a09be3640 15 client.4283.objecter _session_command_op_remove -1 2 2024-04-04T21:02:27.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.108+0000 7f3a09be3640 20 client.4283.objecter get_session s=0x7f39e4072c20 osd=5 3 2024-04-04T21:02:27.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.108+0000 7f3a09be3640 15 client.4283.objecter _session_command_op_assign 5 2 2024-04-04T21:02:27.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.108+0000 7f3a09be3640 20 client.4283.objecter put_session s=0x7f39e4072c20 osd=5 4 2024-04-04T21:02:27.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.108+0000 7f3a09be3640 10 client.4283.objecter _send_command 2 2024-04-04T21:02:27.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.108+0000 7f3a09be3640 1 -- 172.21.15.106:0/3436398859 --> [v2:172.21.15.203:6800/696438187,v1:172.21.15.203:6801/696438187] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3a0406de30 con 0x7f39e40732f0 2024-04-04T21:02:27.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.114+0000 7f3a017fa640 1 -- 172.21.15.106:0/3436398859 <== osd.5 v2:172.21.15.203:6800/696438187 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3a0406de30 con 0x7f39e40732f0 2024-04-04T21:02:27.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.114+0000 7f3a017fa640 10 client.4283.objecter ms_dispatch 0x7f3a04000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:02:27.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.114+0000 7f3a017fa640 10 client.4283.objecter _finish_command 2 = system:0 2024-04-04T21:02:27.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.114+0000 7f3a017fa640 20 client.4283.objecter put_session s=0x7f39e4072c20 osd=5 4 2024-04-04T21:02:27.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.114+0000 7f3a017fa640 15 client.4283.objecter _session_command_op_remove 5 2 2024-04-04T21:02:27.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f03eeffd640 1 -- 172.21.15.106:0/3245008356 <== osd.0 v2:172.21.15.106:6818/2744620362 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f03f4008e50 con 0x7f03e00732f0 2024-04-04T21:02:27.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f03eeffd640 10 client.4324.objecter ms_dispatch 0x7f0400000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:02:27.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f03eeffd640 10 client.4324.objecter _finish_command 1 = system:0 2024-04-04T21:02:27.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f03eeffd640 20 client.4324.objecter put_session s=0x7f03e0072c20 osd=0 4 2024-04-04T21:02:27.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f03eeffd640 15 client.4324.objecter _session_command_op_remove 0 1 2024-04-04T21:02:27.169 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 10 librados: watch_flush enter 2024-04-04T21:02:27.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 10 librados: watch_flush exit 2024-04-04T21:02:27.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 10 client.4283.objecter close_session for osd.5 2024-04-04T21:02:27.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 1 -- 172.21.15.106:0/3436398859 >> [v2:172.21.15.203:6800/696438187,v1:172.21.15.203:6801/696438187] conn(0x7f39e40732f0 msgr2=0x7f39e4075710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 1 --2- 172.21.15.106:0/3436398859 >> [v2:172.21.15.203:6800/696438187,v1:172.21.15.203:6801/696438187] conn(0x7f39e40732f0 0x7f39e4075710 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 20 client.4283.objecter put_session s=0x7f39e4072c20 osd=5 1 2024-04-04T21:02:27.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 20 client.4283.objecter shutdown clearing up homeless session... 2024-04-04T21:02:27.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 10 client.4283.objecter successfully canceled tick 2024-04-04T21:02:27.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 1 -- 172.21.15.106:0/3436398859 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f39e406ceb0 msgr2=0x7f39e406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 1 --2- 172.21.15.106:0/3436398859 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f39e406ceb0 0x7f39e406f370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f39f4001560 tx=0x7f39f4007990 comp rx=0 tx=0).stop 2024-04-04T21:02:27.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 10 monclient: shutdown 2024-04-04T21:02:27.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 20 monclient: shutdown canceling and discarding version request 1 2024-04-04T21:02:27.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.170 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 1 -- 172.21.15.106:0/3436398859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a04143de0 msgr2=0x7f3a04143280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a09be3640 1 --2- 172.21.15.106:0/3436398859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a04143de0 0x7f3a04143280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f39ec02b820 tx=0x7f39ec0046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.115+0000 7f3a093e2640 1 -- 172.21.15.106:0/3436398859 reap_dead start 2024-04-04T21:02:27.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.116+0000 7f3a09be3640 1 -- 172.21.15.106:0/3436398859 shutdown_connections 2024-04-04T21:02:27.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.116+0000 7f3a09be3640 1 -- 172.21.15.106:0/3436398859 >> 172.21.15.106:0/3436398859 conn(0x7f3a040b2850 msgr2=0x7f3a040aec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.116+0000 7f3a09be3640 1 -- 172.21.15.106:0/3436398859 shutdown_connections 2024-04-04T21:02:27.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.116+0000 7f3a09be3640 1 -- 172.21.15.106:0/3436398859 wait complete. 2024-04-04T21:02:27.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.116+0000 7f3a09be3640 1 librados: shutdown 2024-04-04T21:02:27.171 INFO:teuthology.orchestra.run.smithi106.stdout:55834574854 2024-04-04T21:02:27.171 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-04T21:02:27.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.137+0000 7f0404b1c640 10 client.4324.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-04T21:02:27.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.137+0000 7f0404b1c640 15 client.4324.objecter _session_command_op_assign -1 2 2024-04-04T21:02:27.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.137+0000 7f0404b1c640 20 client.4324.objecter _get_session s=0x7f03e0072c20 osd=0 3 2024-04-04T21:02:27.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.137+0000 7f0404b1c640 20 client.4324.objecter put_session s=0x7f03e0072c20 osd=0 3 2024-04-04T21:02:27.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.137+0000 7f0404b1c640 20 client.4324.objecter _get_session s=0x7f03e0072c20 osd=0 3 2024-04-04T21:02:27.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.137+0000 7f0404b1c640 15 client.4324.objecter _session_command_op_remove -1 2 2024-04-04T21:02:27.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.137+0000 7f0404b1c640 20 client.4324.objecter get_session s=0x7f03e0072c20 osd=0 3 2024-04-04T21:02:27.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.137+0000 7f0404b1c640 15 client.4324.objecter _session_command_op_assign 0 2 2024-04-04T21:02:27.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.137+0000 7f0404b1c640 20 client.4324.objecter put_session s=0x7f03e0072c20 osd=0 4 2024-04-04T21:02:27.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.137+0000 7f0404b1c640 10 client.4324.objecter _send_command 2 2024-04-04T21:02:27.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.137+0000 7f0404b1c640 1 -- 172.21.15.106:0/3245008356 --> [v2:172.21.15.106:6818/2744620362,v1:172.21.15.106:6819/2744620362] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f040004f050 con 0x7f03e00732f0 2024-04-04T21:02:27.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.143+0000 7f03eeffd640 1 -- 172.21.15.106:0/3245008356 <== osd.0 v2:172.21.15.106:6818/2744620362 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f040004f050 con 0x7f03e00732f0 2024-04-04T21:02:27.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.143+0000 7f03eeffd640 10 client.4324.objecter ms_dispatch 0x7f0400000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:02:27.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.143+0000 7f03eeffd640 10 client.4324.objecter _finish_command 2 = system:0 2024-04-04T21:02:27.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.143+0000 7f03eeffd640 20 client.4324.objecter put_session s=0x7f03e0072c20 osd=0 4 2024-04-04T21:02:27.174 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.143+0000 7f03eeffd640 15 client.4324.objecter _session_command_op_remove 0 2 2024-04-04T21:02:27.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.143+0000 7f0404b1c640 10 librados: watch_flush enter 2024-04-04T21:02:27.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.143+0000 7f0404b1c640 10 librados: watch_flush exit 2024-04-04T21:02:27.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.143+0000 7f0404b1c640 10 client.4324.objecter close_session for osd.0 2024-04-04T21:02:27.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.143+0000 7f0404b1c640 1 -- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:6818/2744620362,v1:172.21.15.106:6819/2744620362] conn(0x7f03e00732f0 msgr2=0x7f03e0075710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.143+0000 7f0404b1c640 1 --2- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:6818/2744620362,v1:172.21.15.106:6819/2744620362] conn(0x7f03e00732f0 0x7f03e0075710 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.143+0000 7f0404b1c640 20 client.4324.objecter put_session s=0x7f03e0072c20 osd=0 1 2024-04-04T21:02:27.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.143+0000 7f0404b1c640 20 client.4324.objecter shutdown clearing up homeless session... 2024-04-04T21:02:27.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.143+0000 7f0404b1c640 10 client.4324.objecter successfully canceled tick 2024-04-04T21:02:27.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.144+0000 7f0404b1c640 1 -- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f03e006ceb0 msgr2=0x7f03e006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.144+0000 7f0404b1c640 1 --2- 172.21.15.106:0/3245008356 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f03e006ceb0 0x7f03e006f370 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f03f8035000 tx=0x7f03f8030000 comp rx=0 tx=0).stop 2024-04-04T21:02:27.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.144+0000 7f0404b1c640 10 monclient: shutdown 2024-04-04T21:02:27.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.144+0000 7f0404b1c640 20 monclient: shutdown canceling and discarding version request 1 2024-04-04T21:02:27.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.144+0000 7f0404b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.144+0000 7f0404b1c640 1 -- 172.21.15.106:0/3245008356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04001442f0 msgr2=0x7f04001df800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.144+0000 7f0404b1c640 1 --2- 172.21.15.106:0/3245008356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04001442f0 0x7f04001df800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f03f400c610 tx=0x7f03f400cad0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.144+0000 7f03fed9d640 1 -- 172.21.15.106:0/3245008356 reap_dead start 2024-04-04T21:02:27.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.145+0000 7f0404b1c640 1 -- 172.21.15.106:0/3245008356 shutdown_connections 2024-04-04T21:02:27.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.145+0000 7f0404b1c640 1 -- 172.21.15.106:0/3245008356 >> 172.21.15.106:0/3245008356 conn(0x7f04000b8030 msgr2=0x7f0400142b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.145+0000 7f0404b1c640 1 -- 172.21.15.106:0/3245008356 shutdown_connections 2024-04-04T21:02:27.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.145+0000 7f0404b1c640 1 -- 172.21.15.106:0/3245008356 wait complete. 2024-04-04T21:02:27.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.145+0000 7f0404b1c640 1 librados: shutdown 2024-04-04T21:02:27.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.147+0000 7f1992ffd640 1 -- 172.21.15.106:0/1332423717 <== osd.6 v2:172.21.15.203:6816/2758852830 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1998008f30 con 0x7f19840734a0 2024-04-04T21:02:27.176 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.147+0000 7f1992ffd640 10 client.4292.objecter ms_dispatch 0x7f19a4000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:02:27.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.147+0000 7f1992ffd640 10 client.4292.objecter _finish_command 1 = system:0 2024-04-04T21:02:27.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.147+0000 7f1992ffd640 20 client.4292.objecter put_session s=0x7f1984072dd0 osd=6 4 2024-04-04T21:02:27.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.147+0000 7f1992ffd640 15 client.4292.objecter _session_command_op_remove 6 1 2024-04-04T21:02:27.177 INFO:teuthology.orchestra.run.smithi106.stdout:64424509447 2024-04-04T21:02:27.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-04T21:02:27.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.160+0000 7f7706ffd640 1 -- 172.21.15.106:0/1173296054 <== osd.1 v2:172.21.15.106:6826/2321665741 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7700016e70 con 0x7f76f40732f0 2024-04-04T21:02:27.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.160+0000 7f7706ffd640 10 client.4311.objecter ms_dispatch 0x7f7710000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:02:27.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.160+0000 7f7706ffd640 10 client.4311.objecter _finish_command 1 = system:0 2024-04-04T21:02:27.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.160+0000 7f7706ffd640 20 client.4311.objecter put_session s=0x7f76f4072c20 osd=1 4 2024-04-04T21:02:27.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.160+0000 7f7706ffd640 15 client.4311.objecter _session_command_op_remove 1 1 2024-04-04T21:02:27.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.178+0000 7f19a359e640 10 client.4292.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-04T21:02:27.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.178+0000 7f19a359e640 15 client.4292.objecter _session_command_op_assign -1 2 2024-04-04T21:02:27.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.178+0000 7f19a359e640 20 client.4292.objecter _get_session s=0x7f1984072dd0 osd=6 3 2024-04-04T21:02:27.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.178+0000 7f19a359e640 20 client.4292.objecter put_session s=0x7f1984072dd0 osd=6 3 2024-04-04T21:02:27.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.178+0000 7f19a359e640 20 client.4292.objecter _get_session s=0x7f1984072dd0 osd=6 3 2024-04-04T21:02:27.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.178+0000 7f19a359e640 15 client.4292.objecter _session_command_op_remove -1 2 2024-04-04T21:02:27.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.178+0000 7f19a359e640 20 client.4292.objecter get_session s=0x7f1984072dd0 osd=6 3 2024-04-04T21:02:27.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.178+0000 7f19a359e640 15 client.4292.objecter _session_command_op_assign 6 2 2024-04-04T21:02:27.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.178+0000 7f19a359e640 20 client.4292.objecter put_session s=0x7f1984072dd0 osd=6 4 2024-04-04T21:02:27.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.178+0000 7f19a359e640 10 client.4292.objecter _send_command 2 2024-04-04T21:02:27.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.178+0000 7f19a359e640 1 -- 172.21.15.106:0/1332423717 --> [v2:172.21.15.203:6816/2758852830,v1:172.21.15.203:6817/2758852830] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f19a40b13f0 con 0x7f19840734a0 2024-04-04T21:02:27.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.183+0000 7f1992ffd640 1 -- 172.21.15.106:0/1332423717 <== osd.6 v2:172.21.15.203:6816/2758852830 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f19a40b13f0 con 0x7f19840734a0 2024-04-04T21:02:27.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f1992ffd640 10 client.4292.objecter ms_dispatch 0x7f19a4000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:02:27.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f1992ffd640 10 client.4292.objecter _finish_command 2 = system:0 2024-04-04T21:02:27.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f1992ffd640 20 client.4292.objecter put_session s=0x7f1984072dd0 osd=6 4 2024-04-04T21:02:27.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f1992ffd640 15 client.4292.objecter _session_command_op_remove 6 2 2024-04-04T21:02:27.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 10 librados: watch_flush enter 2024-04-04T21:02:27.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 10 librados: watch_flush exit 2024-04-04T21:02:27.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 10 client.4292.objecter close_session for osd.6 2024-04-04T21:02:27.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 1 -- 172.21.15.106:0/1332423717 >> [v2:172.21.15.203:6816/2758852830,v1:172.21.15.203:6817/2758852830] conn(0x7f19840734a0 msgr2=0x7f19840758c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 1 --2- 172.21.15.106:0/1332423717 >> [v2:172.21.15.203:6816/2758852830,v1:172.21.15.203:6817/2758852830] conn(0x7f19840734a0 0x7f19840758c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 20 client.4292.objecter put_session s=0x7f1984072dd0 osd=6 1 2024-04-04T21:02:27.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 20 client.4292.objecter shutdown clearing up homeless session... 2024-04-04T21:02:27.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 10 client.4292.objecter successfully canceled tick 2024-04-04T21:02:27.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 1 -- 172.21.15.106:0/1332423717 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f198406d060 msgr2=0x7f198406f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 1 --2- 172.21.15.106:0/1332423717 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f198406d060 0x7f198406f520 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f19a40b52b0 tx=0x7f1994008040 comp rx=0 tx=0).stop 2024-04-04T21:02:27.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 10 monclient: shutdown 2024-04-04T21:02:27.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 20 monclient: shutdown canceling and discarding version request 1 2024-04-04T21:02:27.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 1 -- 172.21.15.106:0/1332423717 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19a4070840 msgr2=0x7f19a4142d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a359e640 1 --2- 172.21.15.106:0/1332423717 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19a4070840 0x7f19a4142d10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1998000c00 tx=0x7f19980046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.184+0000 7f19a2d9d640 1 -- 172.21.15.106:0/1332423717 reap_dead start 2024-04-04T21:02:27.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.185+0000 7f19a359e640 1 -- 172.21.15.106:0/1332423717 shutdown_connections 2024-04-04T21:02:27.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.185+0000 7f19a359e640 1 -- 172.21.15.106:0/1332423717 >> 172.21.15.106:0/1332423717 conn(0x7f19a40b5e30 msgr2=0x7f19a40b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.186+0000 7f19a359e640 1 -- 172.21.15.106:0/1332423717 shutdown_connections 2024-04-04T21:02:27.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.186+0000 7f19a359e640 1 -- 172.21.15.106:0/1332423717 wait complete. 2024-04-04T21:02:27.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.186+0000 7f19a359e640 1 librados: shutdown 2024-04-04T21:02:27.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.188+0000 7f0d3affd640 1 -- 172.21.15.106:0/3580589784 <== osd.2 v2:172.21.15.106:6810/459340769 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0d3c006e50 con 0x7f0d2c073320 2024-04-04T21:02:27.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.188+0000 7f0d3affd640 10 client.4333.objecter ms_dispatch 0x7f0d4c000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:02:27.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.188+0000 7f0d3affd640 10 client.4333.objecter _finish_command 1 = system:0 2024-04-04T21:02:27.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.188+0000 7f0d3affd640 20 client.4333.objecter put_session s=0x7f0d2c072c50 osd=2 4 2024-04-04T21:02:27.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.188+0000 7f0d3affd640 15 client.4333.objecter _session_command_op_remove 2 1 2024-04-04T21:02:27.190 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.189+0000 7f77183f1640 10 client.4311.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-04T21:02:27.190 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.189+0000 7f77183f1640 15 client.4311.objecter _session_command_op_assign -1 2 2024-04-04T21:02:27.190 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.189+0000 7f77183f1640 20 client.4311.objecter _get_session s=0x7f76f4072c20 osd=1 3 2024-04-04T21:02:27.190 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.189+0000 7f77183f1640 20 client.4311.objecter put_session s=0x7f76f4072c20 osd=1 3 2024-04-04T21:02:27.190 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.189+0000 7f77183f1640 20 client.4311.objecter _get_session s=0x7f76f4072c20 osd=1 3 2024-04-04T21:02:27.190 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.189+0000 7f77183f1640 15 client.4311.objecter _session_command_op_remove -1 2 2024-04-04T21:02:27.190 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.189+0000 7f77183f1640 20 client.4311.objecter get_session s=0x7f76f4072c20 osd=1 3 2024-04-04T21:02:27.191 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.189+0000 7f77183f1640 15 client.4311.objecter _session_command_op_assign 1 2 2024-04-04T21:02:27.191 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.189+0000 7f77183f1640 20 client.4311.objecter put_session s=0x7f76f4072c20 osd=1 4 2024-04-04T21:02:27.191 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.189+0000 7f77183f1640 10 client.4311.objecter _send_command 2 2024-04-04T21:02:27.191 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.189+0000 7f77183f1640 1 -- 172.21.15.106:0/1173296054 --> [v2:172.21.15.106:6826/2321665741,v1:172.21.15.106:6827/2321665741] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f76f8001150 con 0x7f76f40732f0 2024-04-04T21:02:27.193 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.193+0000 7f0d3affd640 1 -- 172.21.15.106:0/3580589784 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d3c0543e0 con 0x7f0d4c06e5a0 2024-04-04T21:02:27.193 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.193+0000 7f0d3affd640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-04-04T21:02:27.194 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.194+0000 7f62c7fff640 1 -- 172.21.15.106:0/2912605036 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62d8056370 con 0x7f62e0143000 2024-04-04T21:02:27.194 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.194+0000 7f62c7fff640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-04-04T21:02:27.204 INFO:teuthology.orchestra.run.smithi106.stdout:64424509449 2024-04-04T21:02:27.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-04T21:02:27.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.212+0000 7f7706ffd640 1 -- 172.21.15.106:0/1173296054 <== osd.1 v2:172.21.15.106:6826/2321665741 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f76f8001150 con 0x7f76f40732f0 2024-04-04T21:02:27.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.213+0000 7f7706ffd640 10 client.4311.objecter ms_dispatch 0x7f7710000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:02:27.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.213+0000 7f7706ffd640 10 client.4311.objecter _finish_command 2 = system:0 2024-04-04T21:02:27.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.213+0000 7f7706ffd640 20 client.4311.objecter put_session s=0x7f76f4072c20 osd=1 4 2024-04-04T21:02:27.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.213+0000 7f7706ffd640 15 client.4311.objecter _session_command_op_remove 1 2 2024-04-04T21:02:27.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.213+0000 7f77183f1640 10 librados: watch_flush enter 2024-04-04T21:02:27.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.213+0000 7f77183f1640 10 librados: watch_flush exit 2024-04-04T21:02:27.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.213+0000 7f77183f1640 10 client.4311.objecter close_session for osd.1 2024-04-04T21:02:27.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.213+0000 7f77183f1640 1 -- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:6826/2321665741,v1:172.21.15.106:6827/2321665741] conn(0x7f76f40732f0 msgr2=0x7f76f4075710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.213+0000 7f77183f1640 1 --2- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:6826/2321665741,v1:172.21.15.106:6827/2321665741] conn(0x7f76f40732f0 0x7f76f4075710 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.213+0000 7f77183f1640 20 client.4311.objecter put_session s=0x7f76f4072c20 osd=1 1 2024-04-04T21:02:27.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.213+0000 7f77183f1640 20 client.4311.objecter shutdown clearing up homeless session... 2024-04-04T21:02:27.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.213+0000 7f77183f1640 10 client.4311.objecter successfully canceled tick 2024-04-04T21:02:27.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f77183f1640 1 -- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f76f406ceb0 msgr2=0x7f76f406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f77183f1640 1 --2- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f76f406ceb0 0x7f76f406f370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7708033d70 tx=0x7f77100aa000 comp rx=0 tx=0).stop 2024-04-04T21:02:27.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f77183f1640 10 monclient: shutdown 2024-04-04T21:02:27.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f77183f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f77183f1640 1 -- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080c82b0 msgr2=0x7f770815d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f77183f1640 1 --2- 172.21.15.106:0/1173296054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080c82b0 0x7f770815d5e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f770000d800 tx=0x7f770000dcc0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f771698e640 1 -- 172.21.15.106:0/1173296054 reap_dead start 2024-04-04T21:02:27.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f77183f1640 1 -- 172.21.15.106:0/1173296054 shutdown_connections 2024-04-04T21:02:27.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f77183f1640 1 -- 172.21.15.106:0/1173296054 >> 172.21.15.106:0/1173296054 conn(0x7f7708036610 msgr2=0x7f770802e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f77183f1640 1 -- 172.21.15.106:0/1173296054 shutdown_connections 2024-04-04T21:02:27.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f77183f1640 1 -- 172.21.15.106:0/1173296054 wait complete. 2024-04-04T21:02:27.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f77183f1640 1 librados: shutdown 2024-04-04T21:02:27.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f0d4b59e640 10 client.4333.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-04T21:02:27.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f0d4b59e640 15 client.4333.objecter _session_command_op_assign -1 2 2024-04-04T21:02:27.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f0d4b59e640 20 client.4333.objecter _get_session s=0x7f0d2c072c50 osd=2 3 2024-04-04T21:02:27.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f0d4b59e640 20 client.4333.objecter put_session s=0x7f0d2c072c50 osd=2 3 2024-04-04T21:02:27.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f0d4b59e640 20 client.4333.objecter _get_session s=0x7f0d2c072c50 osd=2 3 2024-04-04T21:02:27.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f0d4b59e640 15 client.4333.objecter _session_command_op_remove -1 2 2024-04-04T21:02:27.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f0d4b59e640 20 client.4333.objecter get_session s=0x7f0d2c072c50 osd=2 3 2024-04-04T21:02:27.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f0d4b59e640 15 client.4333.objecter _session_command_op_assign 2 2 2024-04-04T21:02:27.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f0d4b59e640 20 client.4333.objecter put_session s=0x7f0d2c072c50 osd=2 4 2024-04-04T21:02:27.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f0d4b59e640 10 client.4333.objecter _send_command 2 2024-04-04T21:02:27.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.214+0000 7f0d4b59e640 1 -- 172.21.15.106:0/3580589784 --> [v2:172.21.15.106:6810/459340769,v1:172.21.15.106:6811/459340769] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0d4c146b40 con 0x7f0d2c073320 2024-04-04T21:02:27.222 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.220+0000 7f62c7fff640 1 -- 172.21.15.106:0/2912605036 <== osd.3 v2:172.21.15.106:6802/873500775 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f62cc075da0 con 0x7f62cc0732d0 2024-04-04T21:02:27.222 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.220+0000 7f62c7fff640 10 client.4327.objecter ms_dispatch 0x7f62e0000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:02:27.222 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.220+0000 7f62c7fff640 10 client.4327.objecter _finish_command 1 = system:0 2024-04-04T21:02:27.222 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.220+0000 7f62c7fff640 20 client.4327.objecter put_session s=0x7f62cc072c00 osd=3 4 2024-04-04T21:02:27.222 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.220+0000 7f62c7fff640 15 client.4327.objecter _session_command_op_remove 3 1 2024-04-04T21:02:27.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.226+0000 7fae89ffb640 1 -- 172.21.15.106:0/843317487 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fae90011cd0 con 0x7fae8c007400 2024-04-04T21:02:27.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.226+0000 7fae89ffb640 1 --2- 172.21.15.106:0/843317487 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fae7806ce60 0x7fae7806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.227+0000 7fae98b4c640 1 --2- 172.21.15.106:0/843317487 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fae7806ce60 0x7fae7806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.227+0000 7fae98b4c640 10 monclient: get_auth_request con 0x7fae7806ce60 auth_method 0 2024-04-04T21:02:27.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.228+0000 7fae98b4c640 1 --2- 172.21.15.106:0/843317487 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fae7806ce60 0x7fae7806f320 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fae8c02ca30 tx=0x7fae9406e030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.228+0000 7fae89ffb640 10 client.4320.objecter ms_handle_connect 0x7fae7806ce60 2024-04-04T21:02:27.233 INFO:teuthology.orchestra.run.smithi106.stdout:85899345929 2024-04-04T21:02:27.233 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-04T21:02:27.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.242+0000 7f62e5da4640 10 client.4327.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-04T21:02:27.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.242+0000 7f62e5da4640 15 client.4327.objecter _session_command_op_assign -1 2 2024-04-04T21:02:27.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.242+0000 7f62e5da4640 20 client.4327.objecter _get_session s=0x7f62cc072c00 osd=3 3 2024-04-04T21:02:27.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.242+0000 7f62e5da4640 20 client.4327.objecter put_session s=0x7f62cc072c00 osd=3 3 2024-04-04T21:02:27.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.242+0000 7f62e5da4640 20 client.4327.objecter _get_session s=0x7f62cc072c00 osd=3 3 2024-04-04T21:02:27.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.242+0000 7f62e5da4640 15 client.4327.objecter _session_command_op_remove -1 2 2024-04-04T21:02:27.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.242+0000 7f62e5da4640 20 client.4327.objecter get_session s=0x7f62cc072c00 osd=3 3 2024-04-04T21:02:27.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.242+0000 7f62e5da4640 15 client.4327.objecter _session_command_op_assign 3 2 2024-04-04T21:02:27.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.242+0000 7f62e5da4640 20 client.4327.objecter put_session s=0x7f62cc072c00 osd=3 4 2024-04-04T21:02:27.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.242+0000 7f62e5da4640 10 client.4327.objecter _send_command 2 2024-04-04T21:02:27.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.242+0000 7f62e5da4640 1 -- 172.21.15.106:0/2912605036 --> [v2:172.21.15.106:6802/873500775,v1:172.21.15.106:6803/873500775] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f62e0072840 con 0x7f62cc0732d0 2024-04-04T21:02:27.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d3affd640 1 -- 172.21.15.106:0/3580589784 <== osd.2 v2:172.21.15.106:6810/459340769 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0d4c146b40 con 0x7f0d2c073320 2024-04-04T21:02:27.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d3affd640 10 client.4333.objecter ms_dispatch 0x7f0d4c000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:02:27.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d3affd640 10 client.4333.objecter _finish_command 2 = system:0 2024-04-04T21:02:27.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d3affd640 20 client.4333.objecter put_session s=0x7f0d2c072c50 osd=2 4 2024-04-04T21:02:27.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d3affd640 15 client.4333.objecter _session_command_op_remove 2 2 2024-04-04T21:02:27.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d4b59e640 10 librados: watch_flush enter 2024-04-04T21:02:27.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d4b59e640 10 librados: watch_flush exit 2024-04-04T21:02:27.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d4b59e640 10 client.4333.objecter close_session for osd.2 2024-04-04T21:02:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d4b59e640 1 -- 172.21.15.106:0/3580589784 >> [v2:172.21.15.106:6810/459340769,v1:172.21.15.106:6811/459340769] conn(0x7f0d2c073320 msgr2=0x7f0d2c075740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d4b59e640 1 --2- 172.21.15.106:0/3580589784 >> [v2:172.21.15.106:6810/459340769,v1:172.21.15.106:6811/459340769] conn(0x7f0d2c073320 0x7f0d2c075740 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d4b59e640 20 client.4333.objecter put_session s=0x7f0d2c072c50 osd=2 1 2024-04-04T21:02:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d4b59e640 20 client.4333.objecter shutdown clearing up homeless session... 2024-04-04T21:02:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d4b59e640 10 client.4333.objecter successfully canceled tick 2024-04-04T21:02:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d4b59e640 1 -- 172.21.15.106:0/3580589784 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f0d2c06cee0 msgr2=0x7f0d2c06f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d4b59e640 1 --2- 172.21.15.106:0/3580589784 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f0d2c06cee0 0x7f0d2c06f3a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0d440012e0 tx=0x7f0d44007d30 comp rx=0 tx=0).stop 2024-04-04T21:02:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d4b59e640 10 monclient: shutdown 2024-04-04T21:02:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d4b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d4b59e640 1 -- 172.21.15.106:0/3580589784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d4c06e5a0 msgr2=0x7f0d4c145fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.251+0000 7f0d4b59e640 1 --2- 172.21.15.106:0/3580589784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d4c06e5a0 0x7f0d4c145fa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00c450 tx=0x7f0d3c00c910 comp rx=0 tx=0).stop 2024-04-04T21:02:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.252+0000 7f0d4ad9d640 1 -- 172.21.15.106:0/3580589784 reap_dead start 2024-04-04T21:02:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.252+0000 7f0d4b59e640 1 -- 172.21.15.106:0/3580589784 shutdown_connections 2024-04-04T21:02:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.252+0000 7f0d4b59e640 1 -- 172.21.15.106:0/3580589784 >> 172.21.15.106:0/3580589784 conn(0x7f0d4c0b3e10 msgr2=0x7f0d4c071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.252+0000 7f0d4b59e640 1 -- 172.21.15.106:0/3580589784 shutdown_connections 2024-04-04T21:02:27.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.252+0000 7f0d4b59e640 1 -- 172.21.15.106:0/3580589784 wait complete. 2024-04-04T21:02:27.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.252+0000 7f0d4b59e640 1 librados: shutdown 2024-04-04T21:02:27.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.253+0000 7f770cff9640 1 -- 172.21.15.106:0/4096817127 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f77100d0f80 con 0x7f77080cfaa0 2024-04-04T21:02:27.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.253+0000 7f770cff9640 10 client.4314.objecter ms_dispatch 0x7f7710000b90 osd_map(26..26 src has 1..26) v4 2024-04-04T21:02:27.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.253+0000 7f770cff9640 3 client.4314.objecter handle_osd_map got epochs [26,26] > 0 2024-04-04T21:02:27.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.253+0000 7f770cff9640 3 client.4314.objecter handle_osd_map decoding full epoch 26 2024-04-04T21:02:27.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.254+0000 7f770cff9640 10 client.4314.objecter checking command 1 2024-04-04T21:02:27.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.254+0000 7f770cff9640 1 --2- 172.21.15.106:0/4096817127 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x7f76f4073210 0x7f76f4075630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.254+0000 7f770cff9640 20 client.4314.objecter _get_session s=0x7f76f4072b40 osd=7 3 2024-04-04T21:02:27.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.254+0000 7f770cff9640 20 client.4314.objecter put_session s=0x7f76f4072b40 osd=7 3 2024-04-04T21:02:27.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.254+0000 7f770cff9640 15 client.4314.objecter _session_command_op_remove -1 1 2024-04-04T21:02:27.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.254+0000 7f770cff9640 20 client.4314.objecter _get_session s=0x7f76f4072b40 osd=7 3 2024-04-04T21:02:27.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.254+0000 7f770cff9640 20 client.4314.objecter get_session s=0x7f76f4072b40 osd=7 3 2024-04-04T21:02:27.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.254+0000 7f770cff9640 15 client.4314.objecter _session_command_op_assign 7 1 2024-04-04T21:02:27.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.254+0000 7f770cff9640 20 client.4314.objecter put_session s=0x7f76f4072b40 osd=7 4 2024-04-04T21:02:27.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.254+0000 7f770cff9640 10 client.4314.objecter _send_command 1 2024-04-04T21:02:27.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.254+0000 7f770cff9640 1 -- 172.21.15.106:0/4096817127 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f771006ee50 con 0x7f76f4073210 2024-04-04T21:02:27.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.254+0000 7f770cff9640 20 client.4314.objecter dump_active .. 0 homeless 2024-04-04T21:02:27.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.258+0000 7f770f7fe640 1 --2- 172.21.15.106:0/4096817127 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x7f76f4073210 0x7f76f4075630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.258+0000 7f770f7fe640 10 monclient: get_auth_request con 0x7f76f4073210 auth_method 0 2024-04-04T21:02:27.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.261+0000 7f770f7fe640 1 --2- 172.21.15.106:0/4096817127 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x7f76f4073210 0x7f76f4075630 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.261+0000 7f770cff9640 10 client.4314.objecter ms_handle_connect 0x7f76f4073210 2024-04-04T21:02:27.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 1 -- 172.21.15.106:0/843317487 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fae900571f0 con 0x7fae8c007400 2024-04-04T21:02:27.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 10 client.4320.objecter ms_dispatch 0x7fae94000b90 osd_map(26..26 src has 1..26) v4 2024-04-04T21:02:27.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 3 client.4320.objecter handle_osd_map got epochs [26,26] > 0 2024-04-04T21:02:27.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 3 client.4320.objecter handle_osd_map decoding full epoch 26 2024-04-04T21:02:27.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 10 client.4320.objecter checking command 1 2024-04-04T21:02:27.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 1 --2- 172.21.15.106:0/843317487 >> [v2:172.21.15.203:6824/2225997929,v1:172.21.15.203:6825/2225997929] conn(0x7fae780732a0 0x7fae780756c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 20 client.4320.objecter _get_session s=0x7fae78072bd0 osd=4 3 2024-04-04T21:02:27.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 20 client.4320.objecter put_session s=0x7fae78072bd0 osd=4 3 2024-04-04T21:02:27.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 15 client.4320.objecter _session_command_op_remove -1 1 2024-04-04T21:02:27.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 20 client.4320.objecter _get_session s=0x7fae78072bd0 osd=4 3 2024-04-04T21:02:27.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 20 client.4320.objecter get_session s=0x7fae78072bd0 osd=4 3 2024-04-04T21:02:27.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 15 client.4320.objecter _session_command_op_assign 4 1 2024-04-04T21:02:27.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 20 client.4320.objecter put_session s=0x7fae78072bd0 osd=4 4 2024-04-04T21:02:27.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 10 client.4320.objecter _send_command 1 2024-04-04T21:02:27.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 1 -- 172.21.15.106:0/843317487 --> [v2:172.21.15.203:6824/2225997929,v1:172.21.15.203:6825/2225997929] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae90014e70 con 0x7fae780732a0 2024-04-04T21:02:27.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 20 client.4320.objecter dump_active .. 0 homeless 2024-04-04T21:02:27.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 1 -- 172.21.15.106:0/843317487 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae90058f20 con 0x7fae8c007400 2024-04-04T21:02:27.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.267+0000 7fae89ffb640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-04-04T21:02:27.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.268+0000 7fae99b4e640 1 --2- 172.21.15.106:0/843317487 >> [v2:172.21.15.203:6824/2225997929,v1:172.21.15.203:6825/2225997929] conn(0x7fae780732a0 0x7fae780756c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.268+0000 7fae99b4e640 10 monclient: get_auth_request con 0x7fae780732a0 auth_method 0 2024-04-04T21:02:27.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.270+0000 7f770cff9640 1 -- 172.21.15.106:0/4096817127 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77100d2cb0 con 0x7f77080cfaa0 2024-04-04T21:02:27.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.270+0000 7f770cff9640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-04-04T21:02:27.272 INFO:teuthology.orchestra.run.smithi106.stdout:85899345929 2024-04-04T21:02:27.272 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-04T21:02:27.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.271+0000 7fae89ffb640 10 client.4320.objecter ms_handle_connect 0x7fae780732a0 2024-04-04T21:02:27.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.271+0000 7fae99b4e640 1 --2- 172.21.15.106:0/843317487 >> [v2:172.21.15.203:6824/2225997929,v1:172.21.15.203:6825/2225997929] conn(0x7fae780732a0 0x7fae780756c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.280+0000 7f62c7fff640 1 -- 172.21.15.106:0/2912605036 <== osd.3 v2:172.21.15.106:6802/873500775 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f62e0072840 con 0x7f62cc0732d0 2024-04-04T21:02:27.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.280+0000 7f62c7fff640 10 client.4327.objecter ms_dispatch 0x7f62e0000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:02:27.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.280+0000 7f62c7fff640 10 client.4327.objecter _finish_command 2 = system:0 2024-04-04T21:02:27.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.280+0000 7f62c7fff640 20 client.4327.objecter put_session s=0x7f62cc072c00 osd=3 4 2024-04-04T21:02:27.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.280+0000 7f62c7fff640 15 client.4327.objecter _session_command_op_remove 3 2 2024-04-04T21:02:27.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.280+0000 7f62e5da4640 10 librados: watch_flush enter 2024-04-04T21:02:27.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.280+0000 7f62e5da4640 10 librados: watch_flush exit 2024-04-04T21:02:27.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.280+0000 7f62e5da4640 10 client.4327.objecter close_session for osd.3 2024-04-04T21:02:27.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.280+0000 7f62e5da4640 1 -- 172.21.15.106:0/2912605036 >> [v2:172.21.15.106:6802/873500775,v1:172.21.15.106:6803/873500775] conn(0x7f62cc0732d0 msgr2=0x7f62cc0756f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.280+0000 7f62e5da4640 1 --2- 172.21.15.106:0/2912605036 >> [v2:172.21.15.106:6802/873500775,v1:172.21.15.106:6803/873500775] conn(0x7f62cc0732d0 0x7f62cc0756f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.280+0000 7f62e5da4640 20 client.4327.objecter put_session s=0x7f62cc072c00 osd=3 1 2024-04-04T21:02:27.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.280+0000 7f62e5da4640 20 client.4327.objecter shutdown clearing up homeless session... 2024-04-04T21:02:27.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.280+0000 7f62e5da4640 10 client.4327.objecter successfully canceled tick 2024-04-04T21:02:27.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.282+0000 7f62e5da4640 1 -- 172.21.15.106:0/2912605036 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f62cc06ce90 msgr2=0x7f62cc06f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.282+0000 7f62e5da4640 1 --2- 172.21.15.106:0/2912605036 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f62cc06ce90 0x7f62cc06f350 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f62d4036000 tx=0x7f62d4030000 comp rx=0 tx=0).stop 2024-04-04T21:02:27.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.282+0000 7f62e5da4640 10 monclient: shutdown 2024-04-04T21:02:27.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.282+0000 7f62e5da4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.282+0000 7f62e5da4640 1 -- 172.21.15.106:0/2912605036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62e0143000 msgr2=0x7f62e0072b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.282+0000 7f62e5da4640 1 --2- 172.21.15.106:0/2912605036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62e0143000 0x7f62e0072b90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f62d8007e40 tx=0x7f62d800e4b0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.282+0000 7f62dffff640 1 -- 172.21.15.106:0/2912605036 reap_dead start 2024-04-04T21:02:27.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.283+0000 7f62e5da4640 1 -- 172.21.15.106:0/2912605036 shutdown_connections 2024-04-04T21:02:27.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.283+0000 7f62e5da4640 1 -- 172.21.15.106:0/2912605036 >> 172.21.15.106:0/2912605036 conn(0x7f62e00b62c0 msgr2=0x7f62e00b0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.283+0000 7f62e5da4640 1 -- 172.21.15.106:0/2912605036 shutdown_connections 2024-04-04T21:02:27.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.283+0000 7f62e5da4640 1 -- 172.21.15.106:0/2912605036 wait complete. 2024-04-04T21:02:27.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.283+0000 7f62e5da4640 1 librados: shutdown 2024-04-04T21:02:27.304 INFO:teuthology.orchestra.run.smithi106.stdout:85899345928 2024-04-04T21:02:27.304 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-04T21:02:27.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.311+0000 7f770cff9640 1 -- 172.21.15.106:0/4096817127 <== osd.7 v2:172.21.15.203:6808/4104640840 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f771006ee50 con 0x7f76f4073210 2024-04-04T21:02:27.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.311+0000 7f770cff9640 10 client.4314.objecter ms_dispatch 0x7f7710000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:02:27.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.311+0000 7f770cff9640 10 client.4314.objecter _finish_command 1 = system:0 2024-04-04T21:02:27.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.311+0000 7f770cff9640 20 client.4314.objecter put_session s=0x7f76f4072b40 osd=7 4 2024-04-04T21:02:27.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.317+0000 7f770cff9640 15 client.4314.objecter _session_command_op_remove 7 1 2024-04-04T21:02:27.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.358+0000 7f77153ee640 10 client.4314.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-04T21:02:27.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.358+0000 7f77153ee640 15 client.4314.objecter _session_command_op_assign -1 2 2024-04-04T21:02:27.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.358+0000 7f77153ee640 20 client.4314.objecter _get_session s=0x7f76f4072b40 osd=7 3 2024-04-04T21:02:27.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.358+0000 7f77153ee640 20 client.4314.objecter put_session s=0x7f76f4072b40 osd=7 3 2024-04-04T21:02:27.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.358+0000 7f77153ee640 20 client.4314.objecter _get_session s=0x7f76f4072b40 osd=7 3 2024-04-04T21:02:27.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.358+0000 7f77153ee640 15 client.4314.objecter _session_command_op_remove -1 2 2024-04-04T21:02:27.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.358+0000 7f77153ee640 20 client.4314.objecter get_session s=0x7f76f4072b40 osd=7 3 2024-04-04T21:02:27.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.358+0000 7f77153ee640 15 client.4314.objecter _session_command_op_assign 7 2 2024-04-04T21:02:27.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.358+0000 7f77153ee640 20 client.4314.objecter put_session s=0x7f76f4072b40 osd=7 4 2024-04-04T21:02:27.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.358+0000 7f77153ee640 10 client.4314.objecter _send_command 2 2024-04-04T21:02:27.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.358+0000 7f77153ee640 1 -- 172.21.15.106:0/4096817127 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f77080ccc00 con 0x7f76f4073210 2024-04-04T21:02:27.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.362+0000 7f2af1271640 10 monclient: get_monmap_and_config 2024-04-04T21:02:27.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.363+0000 7f2af1271640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.363+0000 7f2af1271640 10 monclient: monmap: 2024-04-04T21:02:27.366 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.366 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.366 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:27.364092+0000 2024-04-04T21:02:27.366 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:27.364092+0000 2024-04-04T21:02:27.366 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.366 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.366 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:27.366 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:27.366 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:27.366 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:27.366 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:27.366 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.365+0000 7f2af1271640 1 Processor -- start 2024-04-04T21:02:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.365+0000 7f2af1271640 1 -- start start 2024-04-04T21:02:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.365+0000 7f2af1271640 10 monclient: init 2024-04-04T21:02:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.366+0000 7f2af1271640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.366+0000 7f2af1271640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-04T21:02:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.366+0000 7f2af1271640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2ae4006df0 0x7f2ae402f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.366+0000 7f2af1271640 10 monclient(hunting): picked mon.noname-c con 0x7f2ae4006df0 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.367+0000 7f2af1271640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2ae402f980 0x7f2ae40cc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.367+0000 7f2af1271640 10 monclient(hunting): picked mon.noname-a con 0x7f2ae402f980 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.367+0000 7f2af1271640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ae4030e40 0x7f2ae40cc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.367+0000 7f2af1271640 10 monclient(hunting): picked mon.noname-b con 0x7f2ae4030e40 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.367+0000 7f2af1271640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.367+0000 7f2af1271640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae40071d0 con 0x7f2ae402f980 2024-04-04T21:02:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.367+0000 7f2af1271640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.367+0000 7f2af1271640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae40318e0 con 0x7f2ae4006df0 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.367+0000 7f2af1271640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.367+0000 7f2af1271640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae4031bc0 con 0x7f2ae4030e40 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.367+0000 7f2af1271640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.367+0000 7f2af1271640 10 monclient(hunting): authenticate will time out at 1396.634270s 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.369+0000 7f2aebfff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ae4030e40 0x7f2ae40cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f2aebfff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ae4030e40 0x7f2ae40cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:32824/0 (socket says 172.21.15.106:32824) 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f2aebfff640 1 -- 172.21.15.106:0/3938250408 learned_addr learned my addr 172.21.15.106:0/3938250408 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f2aebfff640 10 monclient(hunting): get_auth_request con 0x7f2ae4030e40 auth_method 0 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f2aebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f2aebfff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f2aebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f2aeaffd640 1 --2- 172.21.15.106:0/3938250408 >> v2:172.21.15.203:3300/0 conn(0x7f2ae4006df0 0x7f2ae402f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f2aeaffd640 10 monclient(hunting): get_auth_request con 0x7f2ae4006df0 auth_method 0 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f2aeaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f2aeaffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f2aeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f770cff9640 1 -- 172.21.15.106:0/4096817127 <== osd.7 v2:172.21.15.203:6808/4104640840 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f77080ccc00 con 0x7f76f4073210 2024-04-04T21:02:27.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f770cff9640 10 client.4314.objecter ms_dispatch 0x7f7710000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:02:27.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f770cff9640 10 client.4314.objecter _finish_command 2 = system:0 2024-04-04T21:02:27.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f770cff9640 20 client.4314.objecter put_session s=0x7f76f4072b40 osd=7 4 2024-04-04T21:02:27.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f770cff9640 15 client.4314.objecter _session_command_op_remove 7 2 2024-04-04T21:02:27.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f2aebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f2aebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.370+0000 7f2aebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.371+0000 7f77153ee640 10 librados: watch_flush enter 2024-04-04T21:02:27.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.371+0000 7f2aeaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.371+0000 7f2aeaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.371+0000 7f2aeaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.371+0000 7f77153ee640 10 librados: watch_flush exit 2024-04-04T21:02:27.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.371+0000 7f77153ee640 10 client.4314.objecter close_session for osd.7 2024-04-04T21:02:27.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.371+0000 7f77153ee640 1 -- 172.21.15.106:0/4096817127 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x7f76f4073210 msgr2=0x7f76f4075630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.371+0000 7f77153ee640 1 --2- 172.21.15.106:0/4096817127 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x7f76f4073210 0x7f76f4075630 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.371+0000 7f77153ee640 20 client.4314.objecter put_session s=0x7f76f4072b40 osd=7 1 2024-04-04T21:02:27.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.371+0000 7f77153ee640 20 client.4314.objecter shutdown clearing up homeless session... 2024-04-04T21:02:27.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.371+0000 7f77153ee640 10 client.4314.objecter successfully canceled tick 2024-04-04T21:02:27.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.372+0000 7f77153ee640 1 -- 172.21.15.106:0/4096817127 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f76f406ce60 msgr2=0x7f76f406f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.372+0000 7f77153ee640 1 --2- 172.21.15.106:0/4096817127 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f76f406ce60 0x7f76f406f320 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7704035000 tx=0x7f7704030000 comp rx=0 tx=0).stop 2024-04-04T21:02:27.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.372+0000 7f77153ee640 10 monclient: shutdown 2024-04-04T21:02:27.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.372+0000 7f77153ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.372+0000 7f77153ee640 1 -- 172.21.15.106:0/4096817127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080cfaa0 msgr2=0x7f77080c9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.372+0000 7f77153ee640 1 --2- 172.21.15.106:0/4096817127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77080cfaa0 0x7f77080c9e80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f771006a920 tx=0x7f771006a010 comp rx=0 tx=0).stop 2024-04-04T21:02:27.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.372+0000 7f7714bed640 1 -- 172.21.15.106:0/4096817127 reap_dead start 2024-04-04T21:02:27.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f77153ee640 1 -- 172.21.15.106:0/4096817127 shutdown_connections 2024-04-04T21:02:27.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f77153ee640 1 -- 172.21.15.106:0/4096817127 >> 172.21.15.106:0/4096817127 conn(0x7f7708006190 msgr2=0x7f7708035930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f77153ee640 1 -- 172.21.15.106:0/4096817127 shutdown_connections 2024-04-04T21:02:27.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f77153ee640 1 -- 172.21.15.106:0/4096817127 wait complete. 2024-04-04T21:02:27.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f77153ee640 1 librados: shutdown 2024-04-04T21:02:27.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.374+0000 7fba236c2640 10 monclient: get_monmap_and_config 2024-04-04T21:02:27.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.374+0000 7fba236c2640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.374+0000 7fba236c2640 10 monclient: monmap: 2024-04-04T21:02:27.378 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.378 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.378 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:27.375189+0000 2024-04-04T21:02:27.378 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:27.375189+0000 2024-04-04T21:02:27.378 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.379 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.379 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:27.379 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:27.379 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:27.379 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:27.379 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:27.379 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f2aeaffd640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2024-04-04T21:02:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f2aeaffd640 1 -- 172.21.15.106:0/3938250408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ae4030e40 msgr2=0x7f2ae40cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f2aeaffd640 1 --2- 172.21.15.106:0/3938250408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ae4030e40 0x7f2ae40cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f2aeaffd640 1 -- 172.21.15.106:0/3938250408 >> v2:172.21.15.106:3300/0 conn(0x7f2ae402f980 msgr2=0x7f2ae40cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f2aeaffd640 1 --2- 172.21.15.106:0/3938250408 >> v2:172.21.15.106:3300/0 conn(0x7f2ae402f980 0x7f2ae40cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f2aeaffd640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f2aeaffd640 1 monclient: found mon.noname-c 2024-04-04T21:02:27.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f2aeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f2aeaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.373+0000 7f2aeaffd640 1 -- 172.21.15.106:0/3938250408 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae4032ef0 con 0x7f2ae4006df0 2024-04-04T21:02:27.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.376+0000 7fba236c2640 1 Processor -- start 2024-04-04T21:02:27.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.379+0000 7f2aeaffd640 1 --2- 172.21.15.106:0/3938250408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae4006df0 0x7f2ae402f440 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4007350 tx=0x7f2ae002b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40e0a829b6ff21a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 1 -- start start 2024-04-04T21:02:27.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 10 monclient: init 2024-04-04T21:02:27.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-04-04T21:02:27.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba1c0a9a50 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:27.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fba1c0a9e30 0x7fba1c142880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 10 monclient(hunting): picked mon.noname-c con 0x7fba1c0a9e30 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:27.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fba1c142dc0 0x7fba1c14cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 10 monclient(hunting): picked mon.noname-a con 0x7fba1c142dc0 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:27.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba1c0ab750 con 0x7fba1c0a9a50 2024-04-04T21:02:27.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1c140120 con 0x7fba1c142dc0 2024-04-04T21:02:27.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1c0aa660 con 0x7fba1c0a9e30 2024-04-04T21:02:27.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.380+0000 7fba236c2640 10 monclient(hunting): authenticate will time out at 1396.648168s 2024-04-04T21:02:27.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.382+0000 7f2ae9ffb640 1 -- 172.21.15.106:0/3938250408 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0038070 con 0x7f2ae4006df0 2024-04-04T21:02:27.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.382+0000 7f2ae9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.382+0000 7f2ae9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:27.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.382+0000 7f2ae9ffb640 10 monclient: dump: 2024-04-04T21:02:27.383 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.383 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.384 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.384 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.384 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.384 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.384 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.384 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.384 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.384 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.382+0000 7f2ae9ffb640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.382+0000 7f2ae9ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.382+0000 7f2ae9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.382+0000 7f2af1271640 5 monclient: authenticate success, global_id 4336 2024-04-04T21:02:27.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.382+0000 7f2af1271640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:27.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.384+0000 7f2ae9ffb640 1 -- 172.21.15.106:0/3938250408 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae002f050 con 0x7f2ae4006df0 2024-04-04T21:02:27.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.384+0000 7f2ae9ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.384+0000 7f2af1271640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:27.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.384+0000 7f2af1271640 4 set_mon_vals no callback set 2024-04-04T21:02:27.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.385+0000 7f2af1271640 10 monclient: shutdown 2024-04-04T21:02:27.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.385+0000 7f2af1271640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.385+0000 7f2af1271640 1 -- 172.21.15.106:0/3938250408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae4006df0 msgr2=0x7f2ae402f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.385+0000 7f2af1271640 1 --2- 172.21.15.106:0/3938250408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae4006df0 0x7f2ae402f440 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4007350 tx=0x7f2ae002b7f0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.389+0000 7f2af1271640 1 -- 172.21.15.106:0/3938250408 shutdown_connections 2024-04-04T21:02:27.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.389+0000 7f2af1271640 1 --2- 172.21.15.106:0/3938250408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ae4030e40 0x7f2ae40cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.389+0000 7f2af1271640 1 --2- 172.21.15.106:0/3938250408 >> v2:172.21.15.106:3300/0 conn(0x7f2ae402f980 0x7f2ae40cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.389+0000 7f2af1271640 1 --2- 172.21.15.106:0/3938250408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae4006df0 0x7f2ae402f440 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.389+0000 7f2af1271640 1 -- 172.21.15.106:0/3938250408 >> 172.21.15.106:0/3938250408 conn(0x7f2ae4006060 msgr2=0x7f2ae4006470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.392+0000 7f2af1271640 1 -- 172.21.15.106:0/3938250408 shutdown_connections 2024-04-04T21:02:27.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.382+0000 7fae89ffb640 1 -- 172.21.15.106:0/843317487 <== osd.4 v2:172.21.15.203:6824/2225997929 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fae90014e70 con 0x7fae780732a0 2024-04-04T21:02:27.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.382+0000 7fae89ffb640 10 client.4320.objecter ms_dispatch 0x7fae94000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:02:27.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.382+0000 7fae89ffb640 10 client.4320.objecter _finish_command 1 = system:0 2024-04-04T21:02:27.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.382+0000 7fae89ffb640 20 client.4320.objecter put_session s=0x7fae78072bd0 osd=4 4 2024-04-04T21:02:27.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.382+0000 7fae89ffb640 15 client.4320.objecter _session_command_op_remove 4 1 2024-04-04T21:02:27.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.389+0000 7fba2145e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fba1c0a9a50 0x7fba1c13f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:41870/0 (socket says 172.21.15.106:41870) 2024-04-04T21:02:27.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.389+0000 7fba20c5d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fba1c0a9e30 0x7fba1c142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.389+0000 7fba2145e640 1 -- 172.21.15.106:0/3796947996 learned_addr learned my addr 172.21.15.106:0/3796947996 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:27.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.389+0000 7fba20c5d640 10 monclient(hunting): get_auth_request con 0x7fba1c0a9e30 auth_method 0 2024-04-04T21:02:27.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.389+0000 7fba20c5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.389+0000 7fba20c5d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.389+0000 7fba20c5d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.392+0000 7fba20c5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.392+0000 7fba20c5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.396 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.392+0000 7fba20c5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.397 INFO:teuthology.orchestra.run.smithi106.stdout:60129542151 2024-04-04T21:02:27.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-04T21:02:27.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.392+0000 7f2af1271640 1 -- 172.21.15.106:0/3938250408 wait complete. 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.393+0000 7f2af1271640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.393+0000 7f2af1271640 10 monclient: monmap: 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.393+0000 7f2af1271640 1 librados: starting msgr at 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.393+0000 7f2af1271640 1 librados: starting objecter 2024-04-04T21:02:27.399 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.393+0000 7f2af1271640 1 Processor -- start 2024-04-04T21:02:27.400 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.393+0000 7f2af1271640 1 -- start start 2024-04-04T21:02:27.400 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.395+0000 7fba20c5d640 10 monclient(hunting): handle_auth_done global_id 4339 payload 274 2024-04-04T21:02:27.400 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.393+0000 7f2af1271640 1 librados: setting wanted keys 2024-04-04T21:02:27.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.395+0000 7fba20c5d640 1 -- 172.21.15.106:0/3796947996 >> v2:172.21.15.106:3300/0 conn(0x7fba1c142dc0 msgr2=0x7fba1c14cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:27.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.395+0000 7fba20c5d640 1 --2- 172.21.15.106:0/3796947996 >> v2:172.21.15.106:3300/0 conn(0x7fba1c142dc0 0x7fba1c14cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.395+0000 7fba20c5d640 1 -- 172.21.15.106:0/3796947996 >> v1:172.21.15.203:6789/0 conn(0x7fba1c0a9a50 legacy=0x7fba1c13f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.395+0000 7fba20c5d640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.395+0000 7fba20c5d640 1 monclient: found mon.noname-c 2024-04-04T21:02:27.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.395+0000 7fba20c5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.395+0000 7fba20c5d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.395+0000 7fba20c5d640 1 -- 172.21.15.106:0/3796947996 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c0aa250 con 0x7fba1c0a9e30 2024-04-04T21:02:27.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.393+0000 7f2af1271640 1 librados: calling monclient init 2024-04-04T21:02:27.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.393+0000 7f2af1271640 10 monclient: init 2024-04-04T21:02:27.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:02:27.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ae4006df0 0x7f2ae40d0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 10 monclient(hunting): picked mon.noname-c con 0x7f2ae4006df0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae402f980 0x7f2ae40ce610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 10 monclient(hunting): picked mon.noname-b con 0x7f2ae402f980 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ae4030e40 0x7f2ae40e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 10 monclient(hunting): picked mon.noname-a con 0x7f2ae4030e40 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae40321d0 con 0x7f2ae4030e40 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae4032050 con 0x7f2ae402f980 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae4034b60 con 0x7f2ae4006df0 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.394+0000 7f2af1271640 10 monclient(hunting): authenticate will time out at 1396.661952s 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.398+0000 7f2aeaffd640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ae4006df0 0x7f2ae40d0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.398+0000 7f2aeaffd640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ae4006df0 0x7f2ae40d0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:32826/0 (socket says 172.21.15.106:32826) 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.398+0000 7f2aeaffd640 1 -- 172.21.15.106:0/3240996473 learned_addr learned my addr 172.21.15.106:0/3240996473 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.398+0000 7f2aeaffd640 10 monclient(hunting): get_auth_request con 0x7f2ae4006df0 auth_method 0 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.398+0000 7f2aeaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.398+0000 7f2aeaffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.398+0000 7f2aeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.399+0000 7f2aea7fc640 1 --2- 172.21.15.106:0/3240996473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae402f980 0x7f2ae40ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.399+0000 7f2aea7fc640 10 monclient(hunting): get_auth_request con 0x7f2ae402f980 auth_method 0 2024-04-04T21:02:27.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.399+0000 7f2aea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.399+0000 7f2aea7fc640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.399+0000 7f2aea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.400+0000 7f2aeaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.400+0000 7f2aeaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.400+0000 7f2aeaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.401+0000 7f2aea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.401+0000 7f2aea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.401+0000 7f2aea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.406+0000 7f2aeaffd640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2024-04-04T21:02:27.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.406+0000 7f2aeaffd640 1 -- 172.21.15.106:0/3240996473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae402f980 msgr2=0x7f2ae40ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.406+0000 7f2aeaffd640 1 --2- 172.21.15.106:0/3240996473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae402f980 0x7f2ae40ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.407+0000 7f2aeaffd640 1 -- 172.21.15.106:0/3240996473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ae4030e40 msgr2=0x7f2ae40e6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.407+0000 7f2aeaffd640 1 --2- 172.21.15.106:0/3240996473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ae4030e40 0x7f2ae40e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.407+0000 7f2aeaffd640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.407+0000 7f2aeaffd640 1 monclient: found mon.noname-c 2024-04-04T21:02:27.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.407+0000 7f2aeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.407+0000 7f2aeaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.407+0000 7f2aeaffd640 1 -- 172.21.15.106:0/3240996473 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae40cf520 con 0x7f2ae4006df0 2024-04-04T21:02:27.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.407+0000 7fba20c5d640 1 --2- 172.21.15.106:0/3796947996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0a9e30 0x7fba1c142880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fba1c06c390 tx=0x7fba0c02b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fbce682d6900d13 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.408+0000 7fdf22a47640 10 monclient: get_monmap_and_config 2024-04-04T21:02:27.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.408+0000 7fdf22a47640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.408+0000 7fdf22a47640 10 monclient: monmap: 2024-04-04T21:02:27.410 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.410 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.411 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:27.409932+0000 2024-04-04T21:02:27.411 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:27.409932+0000 2024-04-04T21:02:27.411 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.411 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.411 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:27.411 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:27.411 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:27.411 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:27.411 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:27.411 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7f2aeaffd640 1 --2- 172.21.15.106:0/3240996473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ae4006df0 0x7f2ae40d0af0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0009890 tx=0x7f2ae00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7f2ad37fe640 10 client.?.objecter ms_handle_connect 0x7f2ae4006df0 2024-04-04T21:02:27.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7f2ad37fe640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.411+0000 7fdf22a47640 1 Processor -- start 2024-04-04T21:02:27.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.411+0000 7fdf22a47640 1 -- start start 2024-04-04T21:02:27.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.411+0000 7fdf22a47640 10 monclient: init 2024-04-04T21:02:27.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.412+0000 7fdf22a47640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.412+0000 7fdf22a47640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-04-04T21:02:27.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.412+0000 7fdf22a47640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdf1c06e810 0x7fdf1c071160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.412+0000 7fdf22a47640 10 monclient(hunting): picked mon.noname-a con 0x7fdf1c06e810 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:27.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.412+0000 7fdf22a47640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf1c06fa70 0x7fdf1c1471f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.412+0000 7fdf22a47640 10 monclient(hunting): picked mon.noname-b con 0x7fdf1c06fa70 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.412+0000 7fdf22a47640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf1c147730 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:27.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.412+0000 7fdf22a47640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf1c070d40 con 0x7fdf1c147730 2024-04-04T21:02:27.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.412+0000 7fdf22a47640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.412+0000 7fdf22a47640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf1c146a80 con 0x7fdf1c06e810 2024-04-04T21:02:27.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.412+0000 7fdf22a47640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.412+0000 7fdf22a47640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c146d60 con 0x7fdf1c06fa70 2024-04-04T21:02:27.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fdf22a47640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fdf22a47640 10 monclient(hunting): authenticate will time out at 1396.680209s 2024-04-04T21:02:27.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba13fff640 1 -- 172.21.15.106:0/3796947996 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c0317c0 con 0x7fba1c0a9e30 2024-04-04T21:02:27.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba13fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:27.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba13fff640 10 monclient: dump: 2024-04-04T21:02:27.415 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.415 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.415 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba13fff640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba13fff640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba13fff640 1 -- 172.21.15.106:0/3796947996 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0c037070 con 0x7fba1c0a9e30 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba236c2640 5 monclient: authenticate success, global_id 4339 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba236c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba13fff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba13fff640 1 -- 172.21.15.106:0/3796947996 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c031bc0 con 0x7fba1c0a9e30 2024-04-04T21:02:27.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba13fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:27.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.413+0000 7fba13fff640 10 monclient: dump: 2024-04-04T21:02:27.417 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.417 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.417 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.417 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.417 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.417 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.417 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.417 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.417 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.417 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.415+0000 7fba236c2640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:27.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.415+0000 7fba236c2640 4 set_mon_vals no callback set 2024-04-04T21:02:27.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.415+0000 7fba236c2640 10 monclient: shutdown 2024-04-04T21:02:27.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.415+0000 7fba236c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.415+0000 7fba236c2640 1 -- 172.21.15.106:0/3796947996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0a9e30 msgr2=0x7fba1c142880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.415+0000 7fba236c2640 1 --2- 172.21.15.106:0/3796947996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0a9e30 0x7fba1c142880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fba1c06c390 tx=0x7fba0c02b860 comp rx=0 tx=0).stop 2024-04-04T21:02:27.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.416+0000 7fdf1b7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf1c06fa70 0x7fdf1c1471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.416+0000 7fdf1b7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf1c06fa70 0x7fdf1c1471f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:32840/0 (socket says 172.21.15.106:32840) 2024-04-04T21:02:27.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.416+0000 7fdf1b7fe640 1 -- 172.21.15.106:0/3080201605 learned_addr learned my addr 172.21.15.106:0/3080201605 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.416+0000 7fdf1b7fe640 10 monclient(hunting): get_auth_request con 0x7fdf1c06fa70 auth_method 0 2024-04-04T21:02:27.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.416+0000 7fdf1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.416+0000 7fdf1b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.416+0000 7fdf1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.417+0000 7fdf1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.417+0000 7fdf1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.417+0000 7fdf1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.417+0000 7fdf1bfff640 1 --2- 172.21.15.106:0/3080201605 >> v2:172.21.15.106:3300/0 conn(0x7fdf1c06e810 0x7fdf1c071160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.417+0000 7fdf1bfff640 10 monclient(hunting): get_auth_request con 0x7fdf1c06e810 auth_method 0 2024-04-04T21:02:27.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.417+0000 7fdf1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.417+0000 7fdf1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.417+0000 7fdf1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.417+0000 7fba236c2640 1 -- 172.21.15.106:0/3796947996 shutdown_connections 2024-04-04T21:02:27.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.417+0000 7fba236c2640 1 --2- 172.21.15.106:0/3796947996 >> v2:172.21.15.106:3300/0 conn(0x7fba1c142dc0 0x7fba1c14cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.417+0000 7fba236c2640 1 --2- 172.21.15.106:0/3796947996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0a9e30 0x7fba1c142880 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.417+0000 7fba236c2640 1 -- 172.21.15.106:0/3796947996 >> 172.21.15.106:0/3796947996 conn(0x7fba1c0b8030 msgr2=0x7fba1c0b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.417+0000 7fba236c2640 1 -- 172.21.15.106:0/3796947996 shutdown_connections 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.417+0000 7fba236c2640 1 -- 172.21.15.106:0/3796947996 wait complete. 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.418+0000 7fba236c2640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.418+0000 7fba236c2640 10 monclient: monmap: 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.419+0000 7fba236c2640 1 librados: starting msgr at 2024-04-04T21:02:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.419+0000 7fba236c2640 1 librados: starting objecter 2024-04-04T21:02:27.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.419+0000 7fba236c2640 1 Processor -- start 2024-04-04T21:02:27.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.419+0000 7fba236c2640 1 -- start start 2024-04-04T21:02:27.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.419+0000 7fba236c2640 1 librados: setting wanted keys 2024-04-04T21:02:27.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.419+0000 7fba236c2640 1 librados: calling monclient init 2024-04-04T21:02:27.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.419+0000 7fba236c2640 10 monclient: init 2024-04-04T21:02:27.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.420+0000 7fdf1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.420+0000 7fdf1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.420+0000 7fdf1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.421+0000 7fba236c2640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.421+0000 7fba236c2640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:02:27.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.421+0000 7fba236c2640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c0a9a50 0x7fba1c143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.421+0000 7fba236c2640 10 monclient(hunting): picked mon.noname-c con 0x7fba1c0a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.421+0000 7fba236c2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0a9e30 0x7fba1c1d7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.421+0000 7fba236c2640 10 monclient(hunting): picked mon.noname-b con 0x7fba1c0a9e30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.421+0000 7fba236c2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba1c142dc0 0x7fba1c1d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.421+0000 7fba236c2640 10 monclient(hunting): picked mon.noname-a con 0x7fba1c142dc0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.420+0000 7fdf1b7fe640 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2024-04-04T21:02:27.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.420+0000 7fdf1b7fe640 1 -- 172.21.15.106:0/3080201605 >> v2:172.21.15.106:3300/0 conn(0x7fdf1c06e810 msgr2=0x7fdf1c071160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:27.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.420+0000 7fdf1b7fe640 1 --2- 172.21.15.106:0/3080201605 >> v2:172.21.15.106:3300/0 conn(0x7fdf1c06e810 0x7fdf1c071160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.420+0000 7fdf1b7fe640 1 -- 172.21.15.106:0/3080201605 >> v1:172.21.15.106:6789/0 conn(0x7fdf1c147730 legacy=0x7fdf1c143e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.420+0000 7fdf1b7fe640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.420+0000 7fdf1b7fe640 1 monclient: found mon.noname-b 2024-04-04T21:02:27.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.420+0000 7fdf1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.420+0000 7fdf1b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.420+0000 7fdf1b7fe640 1 -- 172.21.15.106:0/3080201605 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c144550 con 0x7fdf1c06fa70 2024-04-04T21:02:27.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.421+0000 7fba236c2640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.421+0000 7fba236c2640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c1433b0 con 0x7fba1c142dc0 2024-04-04T21:02:27.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.421+0000 7fba236c2640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.421+0000 7fba236c2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c143230 con 0x7fba1c0a9e30 2024-04-04T21:02:27.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.422+0000 7fba236c2640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.422+0000 7fba236c2640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c14d430 con 0x7fba1c0a9a50 2024-04-04T21:02:27.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.422+0000 7fba236c2640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.422+0000 7fba236c2640 10 monclient(hunting): authenticate will time out at 1396.689414s 2024-04-04T21:02:27.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fdf1b7fe640 1 --2- 172.21.15.106:0/3080201605 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf1c06fa70 0x7fdf1c1471f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdf14009e50 tx=0x7fdf1402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2386ecf25040fbb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fba20c5d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0a9e30 0x7fba1c1d7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fba20c5d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0a9e30 0x7fba1c1d7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50262/0 (socket says 172.21.15.106:50262) 2024-04-04T21:02:27.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fba20c5d640 1 -- 172.21.15.106:0/3931018550 learned_addr learned my addr 172.21.15.106:0/3931018550 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fba20c5d640 10 monclient(hunting): get_auth_request con 0x7fba1c0a9e30 auth_method 0 2024-04-04T21:02:27.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fba20c5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fba20c5d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fba20c5d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.431 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fae9a34f640 10 client.4320.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-04T21:02:27.431 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fae9a34f640 15 client.4320.objecter _session_command_op_assign -1 2 2024-04-04T21:02:27.431 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fae9a34f640 20 client.4320.objecter _get_session s=0x7fae78072bd0 osd=4 3 2024-04-04T21:02:27.431 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fae9a34f640 20 client.4320.objecter put_session s=0x7fae78072bd0 osd=4 3 2024-04-04T21:02:27.431 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fae9a34f640 20 client.4320.objecter _get_session s=0x7fae78072bd0 osd=4 3 2024-04-04T21:02:27.431 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fae9a34f640 15 client.4320.objecter _session_command_op_remove -1 2 2024-04-04T21:02:27.431 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fae9a34f640 20 client.4320.objecter get_session s=0x7fae78072bd0 osd=4 3 2024-04-04T21:02:27.431 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fae9a34f640 15 client.4320.objecter _session_command_op_assign 4 2 2024-04-04T21:02:27.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fae9a34f640 20 client.4320.objecter put_session s=0x7fae78072bd0 osd=4 4 2024-04-04T21:02:27.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fae9a34f640 10 client.4320.objecter _send_command 2 2024-04-04T21:02:27.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fae9a34f640 1 -- 172.21.15.106:0/843317487 --> [v2:172.21.15.203:6824/2225997929,v1:172.21.15.203:6825/2225997929] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fae8c1617d0 con 0x7fae780732a0 2024-04-04T21:02:27.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fba20c5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fba20c5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.425+0000 7fba20c5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.428+0000 7fba2145e640 1 --2- 172.21.15.106:0/3931018550 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c0a9a50 0x7fba1c143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.432 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.430+0000 7fba20c5d640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2024-04-04T21:02:27.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.430+0000 7fba20c5d640 1 -- 172.21.15.106:0/3931018550 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c0a9a50 msgr2=0x7fba1c143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.430+0000 7fba20c5d640 1 --2- 172.21.15.106:0/3931018550 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c0a9a50 0x7fba1c143bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.430+0000 7fba20c5d640 1 -- 172.21.15.106:0/3931018550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba1c142dc0 msgr2=0x7fba1c1d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.430+0000 7fba20c5d640 1 --2- 172.21.15.106:0/3931018550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba1c142dc0 0x7fba1c1d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.430+0000 7fba20c5d640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.430+0000 7fba20c5d640 1 monclient: found mon.noname-b 2024-04-04T21:02:27.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.430+0000 7fba20c5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.430+0000 7fba20c5d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.430+0000 7fba20c5d640 1 -- 172.21.15.106:0/3931018550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c13cc60 con 0x7fba1c0a9e30 2024-04-04T21:02:27.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.429+0000 7fc1357e5640 10 monclient: get_monmap_and_config 2024-04-04T21:02:27.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.429+0000 7fc1357e5640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.429+0000 7fc1357e5640 10 monclient: monmap: 2024-04-04T21:02:27.434 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.434 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.434 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:27.430702+0000 2024-04-04T21:02:27.434 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:27.430702+0000 2024-04-04T21:02:27.435 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.435 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.435 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:27.435 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:27.435 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:27.435 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:27.435 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:27.435 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.431+0000 7fc1357e5640 1 Processor -- start 2024-04-04T21:02:27.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.431+0000 7fc1357e5640 1 -- start start 2024-04-04T21:02:27.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.431+0000 7fc1357e5640 10 monclient: init 2024-04-04T21:02:27.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.432+0000 7fc1357e5640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.432+0000 7fc1357e5640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-04-04T21:02:27.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.432+0000 7fc1357e5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc13006f020 0x7fc13013ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.432+0000 7fc1357e5640 10 monclient(hunting): picked mon.noname-c con 0x7fc13006f020 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:27.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.432+0000 7fc1357e5640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc13006fc70 0x7fc130071720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.432+0000 7fc1357e5640 10 monclient(hunting): picked mon.noname-b con 0x7fc13006fc70 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.432+0000 7fc1357e5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc13006f400 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:27.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.432+0000 7fc1357e5640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc130070a00 con 0x7fc13006f400 2024-04-04T21:02:27.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.432+0000 7fc1357e5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.432+0000 7fc1357e5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc130070ce0 con 0x7fc13006f020 2024-04-04T21:02:27.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.432+0000 7fc1357e5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.432+0000 7fc1357e5640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13006f920 con 0x7fc13006fc70 2024-04-04T21:02:27.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.432+0000 7fc1357e5640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.432+0000 7fc1357e5640 10 monclient(hunting): authenticate will time out at 1396.699993s 2024-04-04T21:02:27.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.433+0000 7fc12f7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc13006fc70 0x7fc130071720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.434+0000 7fba11ffb640 10 client.?.objecter ms_handle_connect 0x7fba1c0a9e30 2024-04-04T21:02:27.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.434+0000 7fba11ffb640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.434+0000 7fba20c5d640 1 --2- 172.21.15.106:0/3931018550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0a9e30 0x7fba1c1d7b60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fba0c034040 tx=0x7fba0c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.434+0000 7fc12ffff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc13006f020 0x7fc13013ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.434+0000 7fc12f7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc13006fc70 0x7fc130071720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:32852/0 (socket says 172.21.15.106:32852) 2024-04-04T21:02:27.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.434+0000 7fc12f7fe640 1 -- 172.21.15.106:0/1742702455 learned_addr learned my addr 172.21.15.106:0/1742702455 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.434+0000 7fc12f7fe640 10 monclient(hunting): get_auth_request con 0x7fc13006fc70 auth_method 0 2024-04-04T21:02:27.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.434+0000 7fc12f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.434+0000 7fc12f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.434+0000 7fc12f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.435+0000 7fc12ffff640 10 monclient(hunting): get_auth_request con 0x7fc13006f020 auth_method 0 2024-04-04T21:02:27.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.435+0000 7fc12ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.435+0000 7fc12ffff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.435+0000 7fc12ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.437+0000 7fc12ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.437+0000 7fc12ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.437+0000 7fc12ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.437+0000 7fc12f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.437+0000 7fc12f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.437+0000 7fc12f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.439+0000 7fc12f7fe640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2024-04-04T21:02:27.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.439+0000 7fc12f7fe640 1 -- 172.21.15.106:0/1742702455 >> v2:172.21.15.203:3300/0 conn(0x7fc13006f020 msgr2=0x7fc13013ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:27.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.439+0000 7fc12f7fe640 1 --2- 172.21.15.106:0/1742702455 >> v2:172.21.15.203:3300/0 conn(0x7fc13006f020 0x7fc13013ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.439+0000 7fc12f7fe640 1 -- 172.21.15.106:0/1742702455 >> v1:172.21.15.106:6789/0 conn(0x7fc13006f400 legacy=0x7fc13013e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:27.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.439+0000 7fc12f7fe640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.439+0000 7fc12f7fe640 1 monclient: found mon.noname-b 2024-04-04T21:02:27.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.439+0000 7fc12f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.439+0000 7fc12f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.439+0000 7fc12f7fe640 1 -- 172.21.15.106:0/1742702455 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1301419f0 con 0x7fc13006fc70 2024-04-04T21:02:27.442 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.441+0000 7fba11ffb640 1 -- 172.21.15.106:0/3931018550 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c032860 con 0x7fba1c0a9e30 2024-04-04T21:02:27.442 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.441+0000 7fba11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.442 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.441+0000 7fba11ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:27.442 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.441+0000 7fba11ffb640 10 monclient: dump: 2024-04-04T21:02:27.442 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.443 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.443 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.443 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.443 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.443 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.443 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.443 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.443 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.443 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.441+0000 7fba11ffb640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.441+0000 7fba11ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.441+0000 7fba11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.441+0000 7fba236c2640 5 monclient: authenticate success, global_id 4345 2024-04-04T21:02:27.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.441+0000 7fba236c2640 10 monclient: _renew_subs 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.441+0000 7fba236c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.441+0000 7fba236c2640 1 -- 172.21.15.106:0/3931018550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1c141f60 con 0x7fba1c0a9e30 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.442+0000 7fba11ffb640 1 -- 172.21.15.106:0/3931018550 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0c032a00 con 0x7fba1c0a9e30 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.442+0000 7fba11ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.442+0000 7fba11ffb640 1 -- 172.21.15.106:0/3931018550 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c032860 con 0x7fba1c0a9e30 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.442+0000 7fba11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.442+0000 7fba11ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.442+0000 7fba11ffb640 10 monclient: dump: 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.444 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.445 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.445 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.445 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.442+0000 7fba12ffd640 4 set_mon_vals no callback set 2024-04-04T21:02:27.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.442+0000 7fba236c2640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.442+0000 7fba236c2640 10 monclient: _renew_subs 2024-04-04T21:02:27.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.442+0000 7fba236c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.442+0000 7fba236c2640 1 -- 172.21.15.106:0/3931018550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1c1441f0 con 0x7fba1c0a9e30 2024-04-04T21:02:27.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.442+0000 7fba236c2640 1 librados: init done 2024-04-04T21:02:27.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.443+0000 7fba236c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.443+0000 7fba236c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.443+0000 7fba236c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.443+0000 7fba236c2640 1 -- 172.21.15.106:0/3931018550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1c13ce30 con 0x7fba1c0a9e30 2024-04-04T21:02:27.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.443+0000 7fae89ffb640 1 -- 172.21.15.106:0/843317487 <== osd.4 v2:172.21.15.203:6824/2225997929 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fae8c1617d0 con 0x7fae780732a0 2024-04-04T21:02:27.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.443+0000 7fae89ffb640 10 client.4320.objecter ms_dispatch 0x7fae94000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:02:27.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.443+0000 7fae89ffb640 10 client.4320.objecter _finish_command 2 = system:0 2024-04-04T21:02:27.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.443+0000 7fae89ffb640 20 client.4320.objecter put_session s=0x7fae78072bd0 osd=4 4 2024-04-04T21:02:27.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.443+0000 7fae89ffb640 15 client.4320.objecter _session_command_op_remove 4 2 2024-04-04T21:02:27.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 10 librados: watch_flush enter 2024-04-04T21:02:27.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 10 librados: watch_flush exit 2024-04-04T21:02:27.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 10 client.4320.objecter close_session for osd.4 2024-04-04T21:02:27.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 1 -- 172.21.15.106:0/843317487 >> [v2:172.21.15.203:6824/2225997929,v1:172.21.15.203:6825/2225997929] conn(0x7fae780732a0 msgr2=0x7fae780756c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 1 --2- 172.21.15.106:0/843317487 >> [v2:172.21.15.203:6824/2225997929,v1:172.21.15.203:6825/2225997929] conn(0x7fae780732a0 0x7fae780756c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 20 client.4320.objecter put_session s=0x7fae78072bd0 osd=4 1 2024-04-04T21:02:27.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 20 client.4320.objecter shutdown clearing up homeless session... 2024-04-04T21:02:27.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 10 client.4320.objecter successfully canceled tick 2024-04-04T21:02:27.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 1 -- 172.21.15.106:0/843317487 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fae7806ce60 msgr2=0x7fae7806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 1 --2- 172.21.15.106:0/843317487 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fae7806ce60 0x7fae7806f320 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fae8c02ca30 tx=0x7fae9406e030 comp rx=0 tx=0).stop 2024-04-04T21:02:27.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 10 monclient: shutdown 2024-04-04T21:02:27.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 1 -- 172.21.15.106:0/843317487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8c007400 msgr2=0x7fae8c0ca1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 1 --2- 172.21.15.106:0/843317487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8c007400 0x7fae8c0ca1a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fae9000c460 tx=0x7fae9000c920 comp rx=0 tx=0).stop 2024-04-04T21:02:27.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae99b4e640 1 -- 172.21.15.106:0/843317487 reap_dead start 2024-04-04T21:02:27.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 1 -- 172.21.15.106:0/843317487 shutdown_connections 2024-04-04T21:02:27.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.444+0000 7fae9a34f640 1 -- 172.21.15.106:0/843317487 >> 172.21.15.106:0/843317487 conn(0x7fae8c006250 msgr2=0x7fae8c0c3150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.445+0000 7fae9a34f640 1 -- 172.21.15.106:0/843317487 shutdown_connections 2024-04-04T21:02:27.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.445+0000 7fae9a34f640 1 -- 172.21.15.106:0/843317487 wait complete. 2024-04-04T21:02:27.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.445+0000 7fae9a34f640 1 librados: shutdown 2024-04-04T21:02:27.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2ad37fe640 1 -- 172.21.15.106:0/3240996473 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0043020 con 0x7f2ae4006df0 2024-04-04T21:02:27.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2ad37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2ad37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:27.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2ad37fe640 10 monclient: dump: 2024-04-04T21:02:27.449 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.450 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2ad37fe640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2ad37fe640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2ad37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2af1271640 5 monclient: authenticate success, global_id 4298 2024-04-04T21:02:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2af1271640 10 monclient: _renew_subs 2024-04-04T21:02:27.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2af1271640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2af1271640 1 -- 172.21.15.106:0/3240996473 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae40d1ce0 con 0x7f2ae4006df0 2024-04-04T21:02:27.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2af1271640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2af1271640 10 monclient: _renew_subs 2024-04-04T21:02:27.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2af1271640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2af1271640 1 -- 172.21.15.106:0/3240996473 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae40e7420 con 0x7f2ae4006df0 2024-04-04T21:02:27.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.446+0000 7f2af1271640 1 librados: init done 2024-04-04T21:02:27.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.447+0000 7fc12f7fe640 1 --2- 172.21.15.106:0/1742702455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc13006fc70 0x7fc130071720 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc1280079e0 tx=0x7fc1280304d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2733c4ed52bcdf04 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.447+0000 7f2af1271640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.447+0000 7f2af1271640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.447+0000 7f2af1271640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.447+0000 7f2af1271640 1 -- 172.21.15.106:0/3240996473 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae40ccec0 con 0x7f2ae4006df0 2024-04-04T21:02:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.452+0000 7fdf1affd640 1 -- 172.21.15.106:0/3080201605 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1402cac0 con 0x7fdf1c06fa70 2024-04-04T21:02:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.453+0000 7fdf1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.453+0000 7fdf1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.453+0000 7fdf1affd640 10 monclient: dump: 2024-04-04T21:02:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.454 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.454 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.454 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.454 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.453+0000 7fdf1affd640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.453+0000 7fdf1affd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.453+0000 7fdf1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.453+0000 7fdf22a47640 5 monclient: authenticate success, global_id 4301 2024-04-04T21:02:27.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.453+0000 7fdf22a47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.457+0000 7fc12effd640 1 -- 172.21.15.106:0/1742702455 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12803e070 con 0x7fc13006fc70 2024-04-04T21:02:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.457+0000 7fc12effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.457+0000 7fc12effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.457+0000 7fc12effd640 10 monclient: dump: 2024-04-04T21:02:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.458 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.457+0000 7fc12effd640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.457+0000 7fc12effd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.457+0000 7fc12effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.457+0000 7fc1357e5640 5 monclient: authenticate success, global_id 4304 2024-04-04T21:02:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.457+0000 7fc1357e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:27.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.459+0000 7fdf1affd640 1 -- 172.21.15.106:0/3080201605 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1402cc60 con 0x7fdf1c06fa70 2024-04-04T21:02:27.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.459+0000 7fdf1affd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.459+0000 7fdf22a47640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.459+0000 7fdf22a47640 4 set_mon_vals no callback set 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.460+0000 7fdf1affd640 1 -- 172.21.15.106:0/3080201605 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf14034610 con 0x7fdf1c06fa70 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.460+0000 7fdf1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.460+0000 7fdf1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.460+0000 7fdf1affd640 10 monclient: dump: 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.461 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.460+0000 7fdf22a47640 10 monclient: shutdown 2024-04-04T21:02:27.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.460+0000 7fdf22a47640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.460+0000 7fdf22a47640 1 -- 172.21.15.106:0/3080201605 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf1c06fa70 msgr2=0x7fdf1c1471f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.460+0000 7fdf22a47640 1 --2- 172.21.15.106:0/3080201605 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf1c06fa70 0x7fdf1c1471f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdf14009e50 tx=0x7fdf1402bf20 comp rx=0 tx=0).stop 2024-04-04T21:02:27.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.460+0000 7fdf22a47640 1 -- 172.21.15.106:0/3080201605 shutdown_connections 2024-04-04T21:02:27.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.460+0000 7fdf22a47640 1 --2- 172.21.15.106:0/3080201605 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf1c06fa70 0x7fdf1c1471f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.460+0000 7fdf22a47640 1 --2- 172.21.15.106:0/3080201605 >> v2:172.21.15.106:3300/0 conn(0x7fdf1c06e810 0x7fdf1c071160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.460+0000 7fdf22a47640 1 -- 172.21.15.106:0/3080201605 >> 172.21.15.106:0/3080201605 conn(0x7fdf1c0b3b90 msgr2=0x7fdf1c0b3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.463 INFO:teuthology.orchestra.run.smithi106.stdout:77309411338 2024-04-04T21:02:27.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.461+0000 7fdf22a47640 1 -- 172.21.15.106:0/3080201605 shutdown_connections 2024-04-04T21:02:27.464 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-04T21:02:27.464 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.461+0000 7fdf22a47640 1 -- 172.21.15.106:0/3080201605 wait complete. 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.461+0000 7fdf22a47640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.462+0000 7fdf22a47640 10 monclient: monmap: 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.462+0000 7fdf22a47640 1 librados: starting msgr at 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.462+0000 7fdf22a47640 1 librados: starting objecter 2024-04-04T21:02:27.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.462+0000 7fdf22a47640 1 Processor -- start 2024-04-04T21:02:27.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.462+0000 7fdf22a47640 1 -- start start 2024-04-04T21:02:27.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.462+0000 7fdf22a47640 1 librados: setting wanted keys 2024-04-04T21:02:27.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.462+0000 7fdf22a47640 1 librados: calling monclient init 2024-04-04T21:02:27.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.462+0000 7fdf22a47640 10 monclient: init 2024-04-04T21:02:27.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:02:27.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf1c06e810 0x7fdf1c13e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 10 monclient(hunting): picked mon.noname-c con 0x7fdf1c06e810 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf1c06fa70 0x7fdf1c12a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 10 monclient(hunting): picked mon.noname-b con 0x7fdf1c06fa70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf1c147730 0x7fdf1c124fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 10 monclient(hunting): picked mon.noname-a con 0x7fdf1c147730 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c151310 con 0x7fdf1c147730 2024-04-04T21:02:27.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c06ee00 con 0x7fdf1c06fa70 2024-04-04T21:02:27.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c151490 con 0x7fdf1c06e810 2024-04-04T21:02:27.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.463+0000 7fdf22a47640 10 monclient(hunting): authenticate will time out at 1396.730916s 2024-04-04T21:02:27.491 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.490+0000 7f5695005640 10 monclient: get_monmap_and_config 2024-04-04T21:02:27.491 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.490+0000 7f5695005640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.491 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.490+0000 7f5695005640 10 monclient: monmap: 2024-04-04T21:02:27.491 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.491 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.491 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:27.492001+0000 2024-04-04T21:02:27.492 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:27.492001+0000 2024-04-04T21:02:27.492 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.492 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.492 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:27.492 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:27.492 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:27.492 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:27.492 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:27.492 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.491+0000 7f5695005640 1 Processor -- start 2024-04-04T21:02:27.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.491+0000 7f5695005640 1 -- start start 2024-04-04T21:02:27.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.491+0000 7f5695005640 10 monclient: init 2024-04-04T21:02:27.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.492+0000 7f5695005640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.492+0000 7f5695005640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-04-04T21:02:27.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.492+0000 7f5695005640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56900a9a50 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:27.494 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.492+0000 7fdf20fe4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf1c147730 0x7fdf1c124fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.494 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.492+0000 7f5695005640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f56900a9e30 0x7f5690142880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.494 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.492+0000 7f5695005640 10 monclient(hunting): picked mon.noname-c con 0x7f56900a9e30 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:27.494 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.492+0000 7f5695005640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5690142dc0 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:27.495 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.492+0000 7f5695005640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56900ab750 con 0x7f5690142dc0 2024-04-04T21:02:27.495 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.492+0000 7f5695005640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5690140120 con 0x7f56900a9a50 2024-04-04T21:02:27.495 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.492+0000 7f5695005640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.495 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.492+0000 7f5695005640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56900aa660 con 0x7f56900a9e30 2024-04-04T21:02:27.495 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.492+0000 7f5695005640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.495 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.492+0000 7f5695005640 10 monclient(hunting): authenticate will time out at 1396.759663s 2024-04-04T21:02:27.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.497+0000 7f2ad37fe640 1 -- 172.21.15.106:0/3240996473 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae002f030 con 0x7f2ae4006df0 2024-04-04T21:02:27.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.497+0000 7f2ad37fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.497+0000 7f2ad37fe640 1 -- 172.21.15.106:0/3240996473 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0038040 con 0x7f2ae4006df0 2024-04-04T21:02:27.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.497+0000 7f2ad37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.497+0000 7f2ad37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.497+0000 7f2ad37fe640 10 monclient: dump: 2024-04-04T21:02:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.498 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.497+0000 7f2ad37fe640 1 -- 172.21.15.106:0/3240996473 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2ae0046020 con 0x7f2ae4006df0 2024-04-04T21:02:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.497+0000 7fc318d68640 10 monclient: get_monmap_and_config 2024-04-04T21:02:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.497+0000 7fc318d68640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.497+0000 7fc318d68640 10 monclient: monmap: 2024-04-04T21:02:27.500 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.500 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.500 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:27.498438+0000 2024-04-04T21:02:27.500 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:27.498438+0000 2024-04-04T21:02:27.500 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.500 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.500 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:27.500 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:27.500 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:27.500 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:27.501 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:27.501 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.498+0000 7fc318d68640 1 Processor -- start 2024-04-04T21:02:27.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.498+0000 7fc318d68640 1 -- start start 2024-04-04T21:02:27.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.498+0000 7fc318d68640 10 monclient: init 2024-04-04T21:02:27.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.498+0000 7f2ad37fe640 1 --2- 172.21.15.106:0/3240996473 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2ad806ce60 0x7f2ad806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.499+0000 7f2aea7fc640 1 --2- 172.21.15.106:0/3240996473 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2ad806ce60 0x7f2ad806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.499+0000 7f2aea7fc640 10 monclient: get_auth_request con 0x7f2ad806ce60 auth_method 0 2024-04-04T21:02:27.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.498+0000 7f2ae8ff9640 4 set_mon_vals no callback set 2024-04-04T21:02:27.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.499+0000 7fc318d68640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.499+0000 7fc318d68640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-04-04T21:02:27.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.499+0000 7fc318d68640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc31406ec20 0x7fc31413b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.499+0000 7fc318d68640 10 monclient(hunting): picked mon.noname-a con 0x7fc31406ec20 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:27.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.500+0000 7fc318d68640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31413ba90 0x7fc3140715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.500+0000 7fc318d68640 10 monclient(hunting): picked mon.noname-b con 0x7fc31413ba90 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.500+0000 7f2ad37fe640 10 client.4298.objecter ms_handle_connect 0x7f2ad806ce60 2024-04-04T21:02:27.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.500+0000 7fc318d68640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc314071b20 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:27.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.500+0000 7fc318d68640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc31406f000 con 0x7fc314071b20 2024-04-04T21:02:27.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.500+0000 7fc318d68640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.500+0000 7fc318d68640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc314070c90 con 0x7fc31406ec20 2024-04-04T21:02:27.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.500+0000 7fc318d68640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.500+0000 7fc318d68640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc314070ee0 con 0x7fc31413ba90 2024-04-04T21:02:27.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.500+0000 7fc318d68640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.500+0000 7fc318d68640 10 monclient(hunting): authenticate will time out at 1396.767672s 2024-04-04T21:02:27.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.500+0000 7f2aea7fc640 1 --2- 172.21.15.106:0/3240996473 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2ad806ce60 0x7f2ad806f320 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2ae40c9ed0 tx=0x7f2adc009060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.514+0000 7f2ad37fe640 1 -- 172.21.15.106:0/3240996473 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2ae0078ec0 con 0x7f2ae4006df0 2024-04-04T21:02:27.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.514+0000 7f2ad37fe640 10 client.4298.objecter ms_dispatch 0x7f2aec000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:27.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.514+0000 7f2ad37fe640 3 client.4298.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:27.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.514+0000 7f2ad37fe640 3 client.4298.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:27.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.514+0000 7f2ad37fe640 20 client.4298.objecter dump_active .. 0 homeless 2024-04-04T21:02:27.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.515+0000 7fc12effd640 1 -- 172.21.15.106:0/1742702455 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc128036480 con 0x7fc13006fc70 2024-04-04T21:02:27.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.515+0000 7fc12effd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.515+0000 7fc1357e5640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.515+0000 7fc1357e5640 4 set_mon_vals no callback set 2024-04-04T21:02:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.516+0000 7fc12effd640 1 -- 172.21.15.106:0/1742702455 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128036850 con 0x7fc13006fc70 2024-04-04T21:02:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.516+0000 7fc12effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.516+0000 7fc12effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.516+0000 7fc12effd640 10 monclient: dump: 2024-04-04T21:02:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.518 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.516+0000 7fc1357e5640 10 monclient: shutdown 2024-04-04T21:02:27.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.516+0000 7fc1357e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.516+0000 7fc1357e5640 1 -- 172.21.15.106:0/1742702455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc13006fc70 msgr2=0x7fc130071720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.516+0000 7fc1357e5640 1 --2- 172.21.15.106:0/1742702455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc13006fc70 0x7fc130071720 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc1280079e0 tx=0x7fc1280304d0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.516+0000 7fc1357e5640 1 -- 172.21.15.106:0/1742702455 shutdown_connections 2024-04-04T21:02:27.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.516+0000 7fc1357e5640 1 --2- 172.21.15.106:0/1742702455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc13006fc70 0x7fc130071720 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.516+0000 7fc1357e5640 1 --2- 172.21.15.106:0/1742702455 >> v2:172.21.15.203:3300/0 conn(0x7fc13006f020 0x7fc13013ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.516+0000 7fc1357e5640 1 -- 172.21.15.106:0/1742702455 >> 172.21.15.106:0/1742702455 conn(0x7fc1300b3c30 msgr2=0x7fc1300b4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.516+0000 7fc1357e5640 1 -- 172.21.15.106:0/1742702455 shutdown_connections 2024-04-04T21:02:27.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.516+0000 7fc1357e5640 1 -- 172.21.15.106:0/1742702455 wait complete. 2024-04-04T21:02:27.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.517+0000 7fc1357e5640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.517+0000 7fc1357e5640 10 monclient: monmap: 2024-04-04T21:02:27.520 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.520 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.520 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.520 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.520 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.520 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.520 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.520 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.521 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.521 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.517+0000 7fc1357e5640 1 librados: starting msgr at 2024-04-04T21:02:27.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.517+0000 7fc1357e5640 1 librados: starting objecter 2024-04-04T21:02:27.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.518+0000 7fc1357e5640 1 Processor -- start 2024-04-04T21:02:27.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.518+0000 7fc1357e5640 1 -- start start 2024-04-04T21:02:27.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.518+0000 7fc1357e5640 1 librados: setting wanted keys 2024-04-04T21:02:27.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.518+0000 7fc1357e5640 1 librados: calling monclient init 2024-04-04T21:02:27.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.518+0000 7fc1357e5640 10 monclient: init 2024-04-04T21:02:27.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fc1357e5640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fc1357e5640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-04T21:02:27.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fc1357e5640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc13006f020 0x7fc13013e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fc1357e5640 10 monclient(hunting): picked mon.noname-c con 0x7fc13006f020 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fc1357e5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13006f400 0x7fc130136d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fc1357e5640 10 monclient(hunting): picked mon.noname-a con 0x7fc13006f400 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fc1357e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1300ab100 0x7fc1301372c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fc1357e5640 10 monclient(hunting): picked mon.noname-b con 0x7fc1300ab100 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fc1357e5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fdf20fe4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf1c147730 0x7fdf1c124fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34938/0 (socket says 172.21.15.106:34938) 2024-04-04T21:02:27.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fdf20fe4640 1 -- 172.21.15.106:0/1831757200 learned_addr learned my addr 172.21.15.106:0/1831757200 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fdf20fe4640 10 monclient(hunting): get_auth_request con 0x7fdf1c147730 auth_method 0 2024-04-04T21:02:27.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fdf20fe4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fdf20fe4640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.519+0000 7fdf20fe4640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.520+0000 7fc1357e5640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13013c0d0 con 0x7fc13006f400 2024-04-04T21:02:27.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.520+0000 7fc1357e5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.520+0000 7fc1357e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13013bf50 con 0x7fc1300ab100 2024-04-04T21:02:27.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.520+0000 7fc1357e5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.520+0000 7fc1357e5640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1300b4750 con 0x7fc13006f020 2024-04-04T21:02:27.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.520+0000 7fc1357e5640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.520+0000 7fc1357e5640 10 monclient(hunting): authenticate will time out at 1396.787311s 2024-04-04T21:02:27.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.521+0000 7fdf20fe4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.521+0000 7fdf20fe4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.522+0000 7fdf20fe4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.527+0000 7fdf20fe4640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-04-04T21:02:27.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.527+0000 7fdf20fe4640 1 -- 172.21.15.106:0/1831757200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf1c06e810 msgr2=0x7fdf1c13e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.527+0000 7fdf20fe4640 1 --2- 172.21.15.106:0/1831757200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf1c06e810 0x7fdf1c13e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.527+0000 7fdf20fe4640 1 -- 172.21.15.106:0/1831757200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf1c06fa70 msgr2=0x7fdf1c12a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.527+0000 7fdf20fe4640 1 --2- 172.21.15.106:0/1831757200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf1c06fa70 0x7fdf1c12a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.527+0000 7fdf20fe4640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.527+0000 7fdf20fe4640 1 monclient: found mon.noname-a 2024-04-04T21:02:27.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.527+0000 7fdf20fe4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.528+0000 7fdf20fe4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.528+0000 7fdf20fe4640 1 -- 172.21.15.106:0/1831757200 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c13f800 con 0x7fdf1c147730 2024-04-04T21:02:27.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.531+0000 7fdf20fe4640 1 --2- 172.21.15.106:0/1831757200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf1c147730 0x7fdf1c124fd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdf1000c3f0 tx=0x7fdf1000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.531+0000 7fdf18ff9640 10 client.?.objecter ms_handle_connect 0x7fdf1c147730 2024-04-04T21:02:27.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.531+0000 7fdf18ff9640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.536 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.536+0000 7fc31259c640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc31406ec20 0x7fc31413b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.537+0000 7fc31259c640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc31406ec20 0x7fc31413b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34952/0 (socket says 172.21.15.106:34952) 2024-04-04T21:02:27.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.537+0000 7fc31259c640 1 -- 172.21.15.106:0/409546702 learned_addr learned my addr 172.21.15.106:0/409546702 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.537+0000 7fc31259c640 10 monclient(hunting): get_auth_request con 0x7fc31406ec20 auth_method 0 2024-04-04T21:02:27.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.537+0000 7fc31259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.537+0000 7fc31259c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.537+0000 7fc31259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.537+0000 7fc12f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13006f400 0x7fc130136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.538+0000 7fc31259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.538+0000 7fc31259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.538+0000 7fc31259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.538+0000 7fc12f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13006f400 0x7fc130136d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34956/0 (socket says 172.21.15.106:34956) 2024-04-04T21:02:27.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.538+0000 7fc12f7fe640 1 -- 172.21.15.106:0/2624920338 learned_addr learned my addr 172.21.15.106:0/2624920338 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.538+0000 7fc12f7fe640 10 monclient(hunting): get_auth_request con 0x7fc13006f400 auth_method 0 2024-04-04T21:02:27.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.538+0000 7fc12f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.538+0000 7fc12f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.538+0000 7fc12f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.541+0000 7fc31259c640 10 monclient(hunting): handle_auth_done global_id 4329 payload 274 2024-04-04T21:02:27.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.541+0000 7fc31259c640 1 -- 172.21.15.106:0/409546702 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31413ba90 msgr2=0x7fc3140715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:27.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.541+0000 7fc31259c640 1 --2- 172.21.15.106:0/409546702 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31413ba90 0x7fc3140715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.541+0000 7fc31259c640 1 -- 172.21.15.106:0/409546702 >> v1:172.21.15.106:6789/0 conn(0x7fc314071b20 legacy=0x7fc314141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:27.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.541+0000 7fc31259c640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.541+0000 7fc31259c640 1 monclient: found mon.noname-a 2024-04-04T21:02:27.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.541+0000 7fc31259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.541+0000 7fc31259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.541+0000 7fc31259c640 1 -- 172.21.15.106:0/409546702 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3140711c0 con 0x7fc31406ec20 2024-04-04T21:02:27.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.542+0000 7fc12f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.542+0000 7fc12f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.542+0000 7fc12f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.545+0000 7fc31259c640 1 --2- 172.21.15.106:0/409546702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc31406ec20 0x7fc31413b550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc3040098a0 tx=0x7fc30402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ae2d745d9e3290 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.549 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc2704d6640 10 monclient: get_monmap_and_config 2024-04-04T21:02:27.549 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc2704d6640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.549 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc2704d6640 10 monclient: monmap: 2024-04-04T21:02:27.549 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.550 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.550 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:27.550338+0000 2024-04-04T21:02:27.550 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:27.550338+0000 2024-04-04T21:02:27.550 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.550 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.550 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:27.550 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:27.550 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:27.550 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:27.550 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:27.550 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc2704d6640 1 Processor -- start 2024-04-04T21:02:27.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc12f7fe640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2024-04-04T21:02:27.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc12f7fe640 1 -- 172.21.15.106:0/2624920338 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc13006f020 msgr2=0x7fc13013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc12f7fe640 1 --2- 172.21.15.106:0/2624920338 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc13006f020 0x7fc13013e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc12f7fe640 1 -- 172.21.15.106:0/2624920338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1300ab100 msgr2=0x7fc1301372c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc12f7fe640 1 --2- 172.21.15.106:0/2624920338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1300ab100 0x7fc1301372c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc12f7fe640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc12f7fe640 1 monclient: found mon.noname-a 2024-04-04T21:02:27.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc12f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc12f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc12f7fe640 1 -- 172.21.15.106:0/2624920338 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1300ace30 con 0x7fc13006f400 2024-04-04T21:02:27.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc2704d6640 1 -- start start 2024-04-04T21:02:27.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.549+0000 7fc2704d6640 10 monclient: init 2024-04-04T21:02:27.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.550+0000 7fc2704d6640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.550+0000 7fc2704d6640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-04-04T21:02:27.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.550+0000 7fc2704d6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc2680a9a50 0x7fc2680a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.550+0000 7fc2704d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc2680a9a50 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:27.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.550+0000 7fc2704d6640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc26813f6a0 0x7fc26813fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.550+0000 7fc2704d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc26813f6a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.550+0000 7fc2704d6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc268142880 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:27.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.550+0000 7fc2704d6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc268143420 con 0x7fc268142880 2024-04-04T21:02:27.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.550+0000 7fc2704d6640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.550+0000 7fc2704d6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2680aa370 con 0x7fc2680a9a50 2024-04-04T21:02:27.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.550+0000 7fc2704d6640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.551+0000 7fc2704d6640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2680ab750 con 0x7fc26813f6a0 2024-04-04T21:02:27.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.551+0000 7fc2704d6640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.551+0000 7fc2704d6640 10 monclient(hunting): authenticate will time out at 1396.818224s 2024-04-04T21:02:27.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf18ff9640 1 -- 172.21.15.106:0/1831757200 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100052d0 con 0x7fdf1c147730 2024-04-04T21:02:27.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf18ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:27.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf18ff9640 10 monclient: dump: 2024-04-04T21:02:27.556 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.556 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.556 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.556 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.556 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.556 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.557 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.557 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.557 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.557 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf18ff9640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf18ff9640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf18ff9640 1 -- 172.21.15.106:0/1831757200 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf10005470 con 0x7fdf1c147730 2024-04-04T21:02:27.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf18ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf18ff9640 1 -- 172.21.15.106:0/1831757200 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf10007260 con 0x7fdf1c147730 2024-04-04T21:02:27.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:27.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf18ff9640 10 monclient: dump: 2024-04-04T21:02:27.557 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf22a47640 5 monclient: authenticate success, global_id 4326 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf22a47640 10 monclient: _renew_subs 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf22a47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf22a47640 1 -- 172.21.15.106:0/1831757200 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1c1438f0 con 0x7fdf1c147730 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf1a7fc640 4 set_mon_vals no callback set 2024-04-04T21:02:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf22a47640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf22a47640 10 monclient: _renew_subs 2024-04-04T21:02:27.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf22a47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf22a47640 1 -- 172.21.15.106:0/1831757200 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1c141020 con 0x7fdf1c147730 2024-04-04T21:02:27.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.552+0000 7fdf22a47640 1 librados: init done 2024-04-04T21:02:27.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.553+0000 7fdf22a47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.553+0000 7fdf22a47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.553+0000 7fdf22a47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.553+0000 7fdf22a47640 1 -- 172.21.15.106:0/1831757200 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1c141250 con 0x7fdf1c147730 2024-04-04T21:02:27.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.554+0000 7fc12f7fe640 1 --2- 172.21.15.106:0/2624920338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13006f400 0x7fc130136d80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc128008b40 tx=0x7fc128004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.554+0000 7f568f59e640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f5690142dc0 0x7f569014cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:49526/0 (socket says 172.21.15.106:49526) 2024-04-04T21:02:27.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.554+0000 7f568f59e640 1 -- 172.21.15.106:0/759376720 learned_addr learned my addr 172.21.15.106:0/759376720 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:27.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.554+0000 7fc12cff9640 10 client.?.objecter ms_handle_connect 0x7fc13006f400 2024-04-04T21:02:27.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.554+0000 7fc12cff9640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.602+0000 7f287fae1640 10 monclient: get_monmap_and_config 2024-04-04T21:02:27.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.602+0000 7f287fae1640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.602+0000 7f287fae1640 10 monclient: monmap: 2024-04-04T21:02:27.603 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.603 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.603 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:27.603213+0000 2024-04-04T21:02:27.603 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:27.603213+0000 2024-04-04T21:02:27.603 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.603 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.603 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:27.603 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:27.603 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:27.603 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:27.603 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:27.604 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.602+0000 7f287fae1640 1 Processor -- start 2024-04-04T21:02:27.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.602+0000 7f287fae1640 1 -- start start 2024-04-04T21:02:27.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.602+0000 7f287fae1640 10 monclient: init 2024-04-04T21:02:27.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.603+0000 7f287fae1640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.603+0000 7f287fae1640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-04-04T21:02:27.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.603+0000 7f287fae1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f28780a9b60 0x7f28781430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.603+0000 7f287fae1640 10 monclient(hunting): picked mon.noname-c con 0x7f28780a9b60 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:27.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.603+0000 7f287fae1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28781426f0 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:27.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.603+0000 7f287fae1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f287813f810 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:27.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.603+0000 7f287fae1640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28781435e0 con 0x7f287813f810 2024-04-04T21:02:27.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.603+0000 7f287fae1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28780b8c90 con 0x7f28781426f0 2024-04-04T21:02:27.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.603+0000 7f287fae1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.603+0000 7f287fae1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28780a9f40 con 0x7f28780a9b60 2024-04-04T21:02:27.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.603+0000 7f287fae1640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.605 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.603+0000 7f287fae1640 10 monclient(hunting): authenticate will time out at 1396.870693s 2024-04-04T21:02:27.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.693+0000 7fba11ffb640 1 -- 172.21.15.106:0/3931018550 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fba0c04c060 con 0x7fba1c0a9e30 2024-04-04T21:02:27.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.694+0000 7fba11ffb640 1 --2- 172.21.15.106:0/3931018550 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fba0006d290 0x7fba0006f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.694+0000 7fba2145e640 1 --2- 172.21.15.106:0/3931018550 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fba0006d290 0x7fba0006f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.694+0000 7fba2145e640 10 monclient: get_auth_request con 0x7fba0006d290 auth_method 0 2024-04-04T21:02:27.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.694+0000 7fba2145e640 1 --2- 172.21.15.106:0/3931018550 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fba0006d290 0x7fba0006f750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fba18005880 tx=0x7fba18009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.694+0000 7fba11ffb640 10 client.4345.objecter ms_handle_connect 0x7fba0006d290 2024-04-04T21:02:27.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.699+0000 7fba11ffb640 1 -- 172.21.15.106:0/3931018550 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fba0c032660 con 0x7fba1c0a9e30 2024-04-04T21:02:27.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.699+0000 7fba11ffb640 10 client.4345.objecter ms_dispatch 0x7fba1c000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:27.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.699+0000 7fba11ffb640 3 client.4345.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:27.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.699+0000 7fba11ffb640 3 client.4345.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:27.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.699+0000 7fba11ffb640 20 client.4345.objecter dump_active .. 0 homeless 2024-04-04T21:02:27.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.709+0000 7fba11ffb640 1 -- 172.21.15.106:0/3931018550 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fba0c040070 con 0x7fba1c0a9e30 2024-04-04T21:02:27.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.709+0000 7fba11ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.709+0000 7fba11ffb640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:27.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.754+0000 7f568e59c640 1 --2- 172.21.15.106:0/759376720 >> v2:172.21.15.203:3300/0 conn(0x7f56900a9e30 0x7f5690142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.771+0000 7f568e59c640 10 monclient(hunting): get_auth_request con 0x7f56900a9e30 auth_method 0 2024-04-04T21:02:27.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.771+0000 7f568e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.771+0000 7f568e59c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.771+0000 7f568e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.773+0000 7f568e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.773+0000 7f568e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.773+0000 7f568e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.779+0000 7f568e59c640 10 monclient(hunting): handle_auth_done global_id 4351 payload 274 2024-04-04T21:02:27.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.779+0000 7f568e59c640 1 -- 172.21.15.106:0/759376720 >> v1:172.21.15.203:6789/0 conn(0x7f56900a9a50 legacy=0x7f569013f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:27.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.779+0000 7f568e59c640 1 -- 172.21.15.106:0/759376720 >> v1:172.21.15.106:6789/0 conn(0x7f5690142dc0 legacy=0x7f569014cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.779+0000 7f568e59c640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.779+0000 7f568e59c640 1 monclient: found mon.noname-c 2024-04-04T21:02:27.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.779+0000 7f568e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.779+0000 7f568e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.779+0000 7f568e59c640 1 -- 172.21.15.106:0/759376720 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56900aa250 con 0x7f56900a9e30 2024-04-04T21:02:27.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.782+0000 7fc26e272640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc2680a9a50 0x7fc2680a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.783+0000 7fc26ea73640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc268142880 0x7fc268142d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:41902/0 (socket says 172.21.15.106:41902) 2024-04-04T21:02:27.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.783+0000 7fc26ea73640 1 -- 172.21.15.106:0/4255260192 learned_addr learned my addr 172.21.15.106:0/4255260192 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:27.783 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.783+0000 7f568e59c640 1 --2- 172.21.15.106:0/759376720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56900a9e30 0x7f5690142880 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f56840097a0 tx=0x7f568402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5d63be29095f4dd server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.785+0000 7fc26e272640 10 monclient(hunting): get_auth_request con 0x7fc2680a9a50 auth_method 0 2024-04-04T21:02:27.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.785+0000 7fc26e272640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.785+0000 7fc26e272640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.785+0000 7fc26e272640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.785+0000 7f287d87d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f28780a9b60 0x7f28781430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.785+0000 7f287d87d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f28780a9b60 0x7f28781430a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50302/0 (socket says 172.21.15.106:50302) 2024-04-04T21:02:27.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.785+0000 7f287d87d640 1 -- 172.21.15.106:0/1993656737 learned_addr learned my addr 172.21.15.106:0/1993656737 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.785+0000 7f287d87d640 10 monclient(hunting): get_auth_request con 0x7f28780a9b60 auth_method 0 2024-04-04T21:02:27.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.785+0000 7f287d87d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.785+0000 7f287d87d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.785+0000 7f287d87d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.788+0000 7f287d87d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.788+0000 7f287d87d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.788+0000 7f287d87d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.788+0000 7fc26e272640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.788+0000 7fc26e272640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.789 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.788+0000 7fc26e272640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.791+0000 7fc26e272640 10 monclient(hunting): handle_auth_done global_id 4354 payload 274 2024-04-04T21:02:27.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.791+0000 7fc26e272640 1 -- 172.21.15.106:0/4255260192 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc26813f6a0 msgr2=0x7fc26813fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:27.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.791+0000 7fc26e272640 1 --2- 172.21.15.106:0/4255260192 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc26813f6a0 0x7fc26813fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.791+0000 7fc26e272640 1 -- 172.21.15.106:0/4255260192 >> v1:172.21.15.203:6789/0 conn(0x7fc268142880 legacy=0x7fc268142d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.791+0000 7fc26e272640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.791+0000 7fc26e272640 1 monclient: found mon.noname-c 2024-04-04T21:02:27.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.791+0000 7fc26e272640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.791+0000 7fc26e272640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.791+0000 7fc26e272640 1 -- 172.21.15.106:0/4255260192 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc268140120 con 0x7fc2680a9a50 2024-04-04T21:02:27.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.795+0000 7f287d87d640 10 monclient(hunting): handle_auth_done global_id 4357 payload 274 2024-04-04T21:02:27.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.795+0000 7f287d87d640 1 -- 172.21.15.106:0/1993656737 >> v1:172.21.15.203:6789/0 conn(0x7f28781426f0 legacy=0x7f28780b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:27.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.795+0000 7f287d87d640 1 -- 172.21.15.106:0/1993656737 >> v1:172.21.15.106:6789/0 conn(0x7f287813f810 legacy=0x7f287813fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:27.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.795+0000 7f287d87d640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.795+0000 7f287d87d640 1 monclient: found mon.noname-c 2024-04-04T21:02:27.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.795+0000 7f287d87d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.796+0000 7f287d87d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.796+0000 7f287d87d640 1 -- 172.21.15.106:0/1993656737 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28780b23b0 con 0x7f28780a9b60 2024-04-04T21:02:27.797 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.796+0000 7f2ad37fe640 1 -- 172.21.15.106:0/3240996473 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2ae40ccec0 con 0x7f2ae4006df0 2024-04-04T21:02:27.797 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.796+0000 7f2ad37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.797 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.796+0000 7f2ad37fe640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:27.799 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.799+0000 7fc26e272640 1 --2- 172.21.15.106:0/4255260192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2680a9a50 0x7fc2680a9e30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc2640097a0 tx=0x7fc26402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7169e1e47734f621 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.805 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.805+0000 7f287d87d640 1 --2- 172.21.15.106:0/1993656737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28780a9b60 0x7f28781430a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2874009870 tx=0x7f287402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=119226e25e2578a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.807 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc26d270640 1 -- 172.21.15.106:0/4255260192 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2640315a0 con 0x7fc2680a9a50 2024-04-04T21:02:27.807 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc26d270640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.807 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc26d270640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:27.807 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc26d270640 10 monclient: dump: 2024-04-04T21:02:27.807 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.807 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.807 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.807 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.807 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.807 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.807 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.807 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.807 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.807 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc26d270640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc26d270640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc26d270640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc26d270640 1 -- 172.21.15.106:0/4255260192 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc264037070 con 0x7fc2680a9a50 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc26d270640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc26d270640 1 -- 172.21.15.106:0/4255260192 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc264031960 con 0x7fc2680a9a50 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc26d270640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc26d270640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc26d270640 10 monclient: dump: 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.808 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.809 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.809 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.809 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc2704d6640 5 monclient: authenticate success, global_id 4354 2024-04-04T21:02:27.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc2704d6640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:27.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.806+0000 7fc2704d6640 4 set_mon_vals no callback set 2024-04-04T21:02:27.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.807+0000 7fc2704d6640 10 monclient: shutdown 2024-04-04T21:02:27.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.807+0000 7fc2704d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.807+0000 7fc2704d6640 1 -- 172.21.15.106:0/4255260192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2680a9a50 msgr2=0x7fc2680a9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.807+0000 7fc2704d6640 1 --2- 172.21.15.106:0/4255260192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2680a9a50 0x7fc2680a9e30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc2640097a0 tx=0x7fc26402b840 comp rx=0 tx=0).stop 2024-04-04T21:02:27.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.807+0000 7fc2704d6640 1 -- 172.21.15.106:0/4255260192 shutdown_connections 2024-04-04T21:02:27.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.807+0000 7fc2704d6640 1 --2- 172.21.15.106:0/4255260192 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc26813f6a0 0x7fc26813fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.807+0000 7fc2704d6640 1 --2- 172.21.15.106:0/4255260192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2680a9a50 0x7fc2680a9e30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.807+0000 7fc2704d6640 1 -- 172.21.15.106:0/4255260192 >> 172.21.15.106:0/4255260192 conn(0x7fc2680b8030 msgr2=0x7fc2680b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.807+0000 7fc2704d6640 1 -- 172.21.15.106:0/4255260192 shutdown_connections 2024-04-04T21:02:27.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.807+0000 7fc2704d6640 1 -- 172.21.15.106:0/4255260192 wait complete. 2024-04-04T21:02:27.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.808+0000 7fc2704d6640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.808+0000 7fc2704d6640 10 monclient: monmap: 2024-04-04T21:02:27.811 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.809+0000 7fc2704d6640 1 librados: starting msgr at 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.809+0000 7fc2704d6640 1 librados: starting objecter 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.809+0000 7fc2704d6640 1 Processor -- start 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.809+0000 7fc2704d6640 1 -- start start 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.809+0000 7fc2704d6640 1 librados: setting wanted keys 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.809+0000 7fc2704d6640 1 librados: calling monclient init 2024-04-04T21:02:27.812 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.809+0000 7fc2704d6640 10 monclient: init 2024-04-04T21:02:27.813 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.813 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-04T21:02:27.813 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2680a9a50 0x7fc2681db180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.813 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f568dd9b640 1 -- 172.21.15.106:0/759376720 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56840315a0 con 0x7f56900a9e30 2024-04-04T21:02:27.813 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f568dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f568dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:27.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f568dd9b640 10 monclient: dump: 2024-04-04T21:02:27.814 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.814 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.814 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.814 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.814 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.814 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.814 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.814 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.814 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.814 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f568dd9b640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f568dd9b640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f568dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f568dd9b640 1 -- 172.21.15.106:0/759376720 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5684037070 con 0x7f56900a9e30 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f568dd9b640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f568dd9b640 1 -- 172.21.15.106:0/759376720 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684031960 con 0x7f56900a9e30 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f568dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f568dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f568dd9b640 10 monclient: dump: 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.815 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.816 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.816 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f5695005640 5 monclient: authenticate success, global_id 4351 2024-04-04T21:02:27.816 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f5695005640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:27.816 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7f5695005640 4 set_mon_vals no callback set 2024-04-04T21:02:27.816 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc2680a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.816 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc26813f6a0 0x7fc268135d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.816 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc26813f6a0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.816 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc268142880 0x7fc2681dbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.817 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc268142880 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.817 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.817 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26814d070 con 0x7fc26813f6a0 2024-04-04T21:02:27.817 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.817 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26814cef0 con 0x7fc268142880 2024-04-04T21:02:27.817 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.817 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26814d1f0 con 0x7fc2680a9a50 2024-04-04T21:02:27.817 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.817 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.810+0000 7fc2704d6640 10 monclient(hunting): authenticate will time out at 1397.077758s 2024-04-04T21:02:27.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f5695005640 10 monclient: shutdown 2024-04-04T21:02:27.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f5695005640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f5695005640 1 -- 172.21.15.106:0/759376720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56900a9e30 msgr2=0x7f5690142880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f5695005640 1 --2- 172.21.15.106:0/759376720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56900a9e30 0x7f5690142880 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f56840097a0 tx=0x7f568402b840 comp rx=0 tx=0).stop 2024-04-04T21:02:27.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f5695005640 1 -- 172.21.15.106:0/759376720 shutdown_connections 2024-04-04T21:02:27.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f5695005640 1 --2- 172.21.15.106:0/759376720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56900a9e30 0x7f5690142880 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.818 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f5695005640 1 -- 172.21.15.106:0/759376720 >> 172.21.15.106:0/759376720 conn(0x7f56900b8030 msgr2=0x7f56900b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f5695005640 1 -- 172.21.15.106:0/759376720 shutdown_connections 2024-04-04T21:02:27.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f5695005640 1 -- 172.21.15.106:0/759376720 wait complete. 2024-04-04T21:02:27.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7fc26ea73640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc268142880 0x7fc2681dbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287c87b640 1 -- 172.21.15.106:0/1993656737 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287402e070 con 0x7f28780a9b60 2024-04-04T21:02:27.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287c87b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287c87b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:27.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287c87b640 10 monclient: dump: 2024-04-04T21:02:27.819 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.820 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.820 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.820 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.820 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.820 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.820 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.820 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.820 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.820 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.820 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287c87b640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.820 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287c87b640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.820 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287c87b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.821 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287c87b640 1 -- 172.21.15.106:0/1993656737 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2874037070 con 0x7f28780a9b60 2024-04-04T21:02:27.821 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287c87b640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.821 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287c87b640 1 -- 172.21.15.106:0/1993656737 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874031a00 con 0x7f28780a9b60 2024-04-04T21:02:27.821 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287c87b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.821 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287c87b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:27.821 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287c87b640 10 monclient: dump: 2024-04-04T21:02:27.821 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.821 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.821 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.821 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.821 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.821 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.821 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.821 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.822 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.822 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.822 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287fae1640 5 monclient: authenticate success, global_id 4357 2024-04-04T21:02:27.822 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287fae1640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:27.822 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.811+0000 7f287fae1640 4 set_mon_vals no callback set 2024-04-04T21:02:27.822 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.812+0000 7f5695005640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.822 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.812+0000 7f5695005640 10 monclient: monmap: 2024-04-04T21:02:27.822 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.823 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.823 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.823 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.823 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.823 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.823 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.823 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.825 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.825 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.812+0000 7fc26ea73640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc268142880 0x7fc2681dbca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50306/0 (socket says 172.21.15.106:50306) 2024-04-04T21:02:27.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.812+0000 7fc26ea73640 1 -- 172.21.15.106:0/189408947 learned_addr learned my addr 172.21.15.106:0/189408947 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.812+0000 7fc26ea73640 10 monclient(hunting): get_auth_request con 0x7fc268142880 auth_method 0 2024-04-04T21:02:27.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.812+0000 7fc26ea73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.812+0000 7fc26ea73640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.826 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.812+0000 7fc26ea73640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.812+0000 7f5695005640 1 librados: starting msgr at 2024-04-04T21:02:27.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.812+0000 7f5695005640 1 librados: starting objecter 2024-04-04T21:02:27.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.812+0000 7f287fae1640 10 monclient: shutdown 2024-04-04T21:02:27.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.812+0000 7f287fae1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.812+0000 7f287fae1640 1 -- 172.21.15.106:0/1993656737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28780a9b60 msgr2=0x7f28781430a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.812+0000 7f287fae1640 1 --2- 172.21.15.106:0/1993656737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28780a9b60 0x7f28781430a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2874009870 tx=0x7f287402b880 comp rx=0 tx=0).stop 2024-04-04T21:02:27.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7f287fae1640 1 -- 172.21.15.106:0/1993656737 shutdown_connections 2024-04-04T21:02:27.827 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7f287fae1640 1 --2- 172.21.15.106:0/1993656737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28780a9b60 0x7f28781430a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.828 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7f287fae1640 1 -- 172.21.15.106:0/1993656737 >> 172.21.15.106:0/1993656737 conn(0x7f28780b7ec0 msgr2=0x7f28780b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.828 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7f5695005640 1 Processor -- start 2024-04-04T21:02:27.828 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7f5695005640 1 -- start start 2024-04-04T21:02:27.828 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7f5695005640 1 librados: setting wanted keys 2024-04-04T21:02:27.828 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7f5695005640 1 librados: calling monclient init 2024-04-04T21:02:27.828 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7f5695005640 10 monclient: init 2024-04-04T21:02:27.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7f287fae1640 1 -- 172.21.15.106:0/1993656737 shutdown_connections 2024-04-04T21:02:27.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7f287fae1640 1 -- 172.21.15.106:0/1993656737 wait complete. 2024-04-04T21:02:27.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7f5695005640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7fc26ea73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7fc26ea73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7fc26ea73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7f5695005640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:02:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.813+0000 7f5695005640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56900a9a50 0x7f5690137cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f5695005640 10 monclient(hunting): picked mon.noname-c con 0x7f56900a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f5695005640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56900a9e30 0x7f5690143960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f5695005640 10 monclient(hunting): picked mon.noname-b con 0x7f56900a9e30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f5695005640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5690142dc0 0x7f5690143ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f5695005640 10 monclient(hunting): picked mon.noname-a con 0x7f5690142dc0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f5695005640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f5695005640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56901433b0 con 0x7f5690142dc0 2024-04-04T21:02:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f5695005640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f5695005640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5690143230 con 0x7f56900a9e30 2024-04-04T21:02:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f5695005640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f5695005640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569014e610 con 0x7f56900a9a50 2024-04-04T21:02:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f5695005640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f5695005640 10 monclient(hunting): authenticate will time out at 1397.081447s 2024-04-04T21:02:27.834 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f287fae1640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.835 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f287fae1640 10 monclient: monmap: 2024-04-04T21:02:27.835 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.835 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.835 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.835 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.835 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.835 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.835 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.835 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.836 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.836 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f287fae1640 1 librados: starting msgr at 2024-04-04T21:02:27.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.814+0000 7f287fae1640 1 librados: starting objecter 2024-04-04T21:02:27.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.815+0000 7f287fae1640 1 Processor -- start 2024-04-04T21:02:27.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.815+0000 7f287fae1640 1 -- start start 2024-04-04T21:02:27.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.815+0000 7f287fae1640 1 librados: setting wanted keys 2024-04-04T21:02:27.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.815+0000 7f287fae1640 1 librados: calling monclient init 2024-04-04T21:02:27.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.815+0000 7f287fae1640 10 monclient: init 2024-04-04T21:02:27.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-04T21:02:27.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28780a9b60 0x7f2878138e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 10 monclient(hunting): picked mon.noname-b con 0x7f28780a9b60 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f287813f810 0x7f2878134ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 10 monclient(hunting): picked mon.noname-c con 0x7f287813f810 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2878137290 0x7f28781398a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 10 monclient(hunting): picked mon.noname-a con 0x7f2878137290 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2878142c10 con 0x7f2878137290 2024-04-04T21:02:27.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287814e220 con 0x7f28780a9b60 2024-04-04T21:02:27.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287814e3a0 con 0x7f287813f810 2024-04-04T21:02:27.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.816+0000 7f287fae1640 10 monclient(hunting): authenticate will time out at 1397.083989s 2024-04-04T21:02:27.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.817+0000 7fc26ea73640 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2024-04-04T21:02:27.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.817+0000 7fc26ea73640 1 -- 172.21.15.106:0/189408947 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2680a9a50 msgr2=0x7fc2681db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.817+0000 7fc26ea73640 1 --2- 172.21.15.106:0/189408947 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2680a9a50 0x7fc2681db180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.817+0000 7fc26ea73640 1 -- 172.21.15.106:0/189408947 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc26813f6a0 msgr2=0x7fc268135d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.817+0000 7fc26ea73640 1 --2- 172.21.15.106:0/189408947 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc26813f6a0 0x7fc268135d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.817+0000 7fc26ea73640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.817+0000 7fc26ea73640 1 monclient: found mon.noname-b 2024-04-04T21:02:27.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.817+0000 7fc26ea73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.842 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.817+0000 7fc26ea73640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.842 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.817+0000 7fc26ea73640 1 -- 172.21.15.106:0/189408947 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26813cb70 con 0x7fc268142880 2024-04-04T21:02:27.842 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.819+0000 7f568e59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56900a9e30 0x7f5690143960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.842 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.820+0000 7f568e59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56900a9e30 0x7f5690143960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50310/0 (socket says 172.21.15.106:50310) 2024-04-04T21:02:27.842 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.820+0000 7f568e59c640 1 -- 172.21.15.106:0/1442672833 learned_addr learned my addr 172.21.15.106:0/1442672833 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.843 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.820+0000 7f568e59c640 10 monclient(hunting): get_auth_request con 0x7f56900a9e30 auth_method 0 2024-04-04T21:02:27.843 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.820+0000 7f568e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.843 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.820+0000 7f568e59c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.843 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.820+0000 7f568e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.843 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.820+0000 7fc26ea73640 1 --2- 172.21.15.106:0/189408947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc268142880 0x7fc2681dbca0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc26000ca20 tx=0x7fc26000cee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.843 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.820+0000 7fc25effd640 10 client.?.objecter ms_handle_connect 0x7fc268142880 2024-04-04T21:02:27.843 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.820+0000 7fc25effd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.843 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.821+0000 7f568e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.821+0000 7f568e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.821+0000 7f568e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.821+0000 7f287d87d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28780a9b60 0x7f2878138e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.825+0000 7f568e59c640 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2024-04-04T21:02:27.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.825+0000 7f568e59c640 1 -- 172.21.15.106:0/1442672833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56900a9a50 msgr2=0x7f5690137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.825+0000 7f568e59c640 1 --2- 172.21.15.106:0/1442672833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56900a9a50 0x7f5690137cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.825+0000 7f568e59c640 1 -- 172.21.15.106:0/1442672833 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5690142dc0 msgr2=0x7f5690143ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.825+0000 7f568e59c640 1 --2- 172.21.15.106:0/1442672833 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5690142dc0 0x7f5690143ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.825+0000 7f568e59c640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.825+0000 7f568e59c640 1 monclient: found mon.noname-b 2024-04-04T21:02:27.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.825+0000 7f568e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.825+0000 7f568e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.825+0000 7f568e59c640 1 -- 172.21.15.106:0/1442672833 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5690145c60 con 0x7f56900a9e30 2024-04-04T21:02:27.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.830+0000 7f287d87d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28780a9b60 0x7f2878138e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50324/0 (socket says 172.21.15.106:50324) 2024-04-04T21:02:27.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.830+0000 7f287d87d640 1 -- 172.21.15.106:0/1279842228 learned_addr learned my addr 172.21.15.106:0/1279842228 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.830+0000 7f287d87d640 10 monclient(hunting): get_auth_request con 0x7f28780a9b60 auth_method 0 2024-04-04T21:02:27.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.830+0000 7f287d87d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.830+0000 7f287d87d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.830+0000 7f287d87d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.831+0000 7f568e59c640 1 --2- 172.21.15.106:0/1442672833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56900a9e30 0x7f5690143960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f56840098c0 tx=0x7f56840046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.831+0000 7f56777fe640 10 client.?.objecter ms_handle_connect 0x7f56900a9e30 2024-04-04T21:02:27.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.831+0000 7f56777fe640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc25effd640 1 -- 172.21.15.106:0/189408947 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260017070 con 0x7fc268142880 2024-04-04T21:02:27.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc25effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc25effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:27.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc25effd640 10 monclient: dump: 2024-04-04T21:02:27.847 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.847 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc25effd640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc25effd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc25effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc25effd640 1 -- 172.21.15.106:0/189408947 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26000abd0 con 0x7fc268142880 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc25effd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc25effd640 1 -- 172.21.15.106:0/189408947 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260012400 con 0x7fc268142880 2024-04-04T21:02:27.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc2704d6640 5 monclient: authenticate success, global_id 4363 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc2704d6640 10 monclient: _renew_subs 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc2704d6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc2704d6640 1 -- 172.21.15.106:0/189408947 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc268143bb0 con 0x7fc268142880 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.832+0000 7fc26ca6f640 4 set_mon_vals no callback set 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.833+0000 7fc25effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.833+0000 7fc25effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.833+0000 7fc25effd640 10 monclient: dump: 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.849 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.850 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.850 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.833+0000 7fc2704d6640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.833+0000 7fc2704d6640 10 monclient: _renew_subs 2024-04-04T21:02:27.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.833+0000 7fc2704d6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.833+0000 7fc2704d6640 1 -- 172.21.15.106:0/189408947 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2681403e0 con 0x7fc268142880 2024-04-04T21:02:27.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.833+0000 7fc2704d6640 1 librados: init done 2024-04-04T21:02:27.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.833+0000 7f287d87d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.851 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.833+0000 7f287d87d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.851 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.833+0000 7f287d87d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.851 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.834+0000 7fc2704d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.851 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.834+0000 7fc2704d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.851 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.834+0000 7fc2704d6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.851 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.834+0000 7fc2704d6640 1 -- 172.21.15.106:0/189408947 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26804f050 con 0x7fc268142880 2024-04-04T21:02:27.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f287d87d640 10 monclient(hunting): handle_auth_done global_id 4369 payload 824 2024-04-04T21:02:27.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f287d87d640 1 -- 172.21.15.106:0/1279842228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f287813f810 msgr2=0x7f2878134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f287d87d640 1 --2- 172.21.15.106:0/1279842228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f287813f810 0x7f2878134ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f287d87d640 1 -- 172.21.15.106:0/1279842228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2878137290 msgr2=0x7f28781398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f287d87d640 1 --2- 172.21.15.106:0/1279842228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2878137290 0x7f28781398a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f287d87d640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f287d87d640 1 monclient: found mon.noname-b 2024-04-04T21:02:27.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f287d87d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f287d87d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f287d87d640 1 -- 172.21.15.106:0/1279842228 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287813acb0 con 0x7f28780a9b60 2024-04-04T21:02:27.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f56777fe640 1 -- 172.21.15.106:0/1442672833 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684032840 con 0x7f56900a9e30 2024-04-04T21:02:27.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f56777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f56777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:27.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f56777fe640 10 monclient: dump: 2024-04-04T21:02:27.856 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.856 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.856 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.856 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.856 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.856 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.856 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.856 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.856 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.856 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f56777fe640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f56777fe640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f56777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f56777fe640 1 -- 172.21.15.106:0/1442672833 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5684032a00 con 0x7f56900a9e30 2024-04-04T21:02:27.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f56777fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f56777fe640 1 -- 172.21.15.106:0/1442672833 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684032df0 con 0x7f56900a9e30 2024-04-04T21:02:27.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f56777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f56777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:27.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f56777fe640 10 monclient: dump: 2024-04-04T21:02:27.857 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.857 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.857 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.857 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.857 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.857 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.857 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.840+0000 7f568d59a640 4 set_mon_vals no callback set 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.841+0000 7f5695005640 5 monclient: authenticate success, global_id 4366 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.841+0000 7f5695005640 10 monclient: _renew_subs 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.841+0000 7f5695005640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.841+0000 7f5695005640 1 -- 172.21.15.106:0/1442672833 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56901463b0 con 0x7f56900a9e30 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.841+0000 7f5695005640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.841+0000 7f5695005640 10 monclient: _renew_subs 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.841+0000 7f5695005640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.841+0000 7f5695005640 1 -- 172.21.15.106:0/1442672833 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5690144ae0 con 0x7f56900a9e30 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.841+0000 7f5695005640 1 librados: init done 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.842+0000 7f5695005640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.842+0000 7f5695005640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.842+0000 7f5695005640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.842+0000 7f5695005640 1 -- 172.21.15.106:0/1442672833 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569004f050 con 0x7f56900a9e30 2024-04-04T21:02:27.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.843+0000 7f287d87d640 1 --2- 172.21.15.106:0/1279842228 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28780a9b60 0x7f2878138e10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2874000c00 tx=0x7f28740046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.843+0000 7f286e7fc640 10 client.?.objecter ms_handle_connect 0x7f28780a9b60 2024-04-04T21:02:27.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.843+0000 7f286e7fc640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.862+0000 7fdf18ff9640 1 -- 172.21.15.106:0/1831757200 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdf10007400 con 0x7fdf1c147730 2024-04-04T21:02:27.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.863+0000 7fdf18ff9640 1 --2- 172.21.15.106:0/1831757200 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdf0806ceb0 0x7fdf0806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.863+0000 7fdf1bfff640 1 --2- 172.21.15.106:0/1831757200 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdf0806ceb0 0x7fdf0806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.863+0000 7fdf1bfff640 10 monclient: get_auth_request con 0x7fdf0806ceb0 auth_method 0 2024-04-04T21:02:27.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.863+0000 7fdf1bfff640 1 --2- 172.21.15.106:0/1831757200 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdf0806ceb0 0x7fdf0806f370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c006a90 tx=0x7fdf0c006a20 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.863+0000 7fdf18ff9640 10 client.4326.objecter ms_handle_connect 0x7fdf0806ceb0 2024-04-04T21:02:27.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.878+0000 7fdf18ff9640 1 -- 172.21.15.106:0/1831757200 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdf10056670 con 0x7fdf1c147730 2024-04-04T21:02:27.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.878+0000 7fdf18ff9640 10 client.4326.objecter ms_dispatch 0x7fdf1c000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:27.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.878+0000 7fdf18ff9640 3 client.4326.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:27.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.878+0000 7fdf18ff9640 3 client.4326.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:27.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.878+0000 7fdf18ff9640 20 client.4326.objecter dump_active .. 0 homeless 2024-04-04T21:02:27.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.879+0000 7fdf18ff9640 1 -- 172.21.15.106:0/1831757200 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdf10058cd0 con 0x7fdf1c147730 2024-04-04T21:02:27.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.879+0000 7fdf18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.879+0000 7fdf18ff9640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:27.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.881+0000 7fc12cff9640 1 -- 172.21.15.106:0/2624920338 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128048070 con 0x7fc13006f400 2024-04-04T21:02:27.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.881+0000 7fc12cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.881+0000 7fc12cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:27.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.881+0000 7fc12cff9640 10 monclient: dump: 2024-04-04T21:02:27.882 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.882 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.882 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.882 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.882 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.882 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.881+0000 7fc12cff9640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.881+0000 7fc12cff9640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.881+0000 7fc12cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.881+0000 7fc12cff9640 1 -- 172.21.15.106:0/2624920338 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc128043670 con 0x7fc13006f400 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.881+0000 7fc12cff9640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.881+0000 7fc12cff9640 1 -- 172.21.15.106:0/2624920338 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12803e040 con 0x7fc13006f400 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.881+0000 7fc12cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.881+0000 7fc12cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.881+0000 7fc12cff9640 10 monclient: dump: 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.883 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.884 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.884 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.884 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.884 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.884 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.884 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.884 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.884 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.881+0000 7fc12e7fc640 4 set_mon_vals no callback set 2024-04-04T21:02:27.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.882+0000 7fc1357e5640 5 monclient: authenticate success, global_id 4332 2024-04-04T21:02:27.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.882+0000 7fc1357e5640 10 monclient: _renew_subs 2024-04-04T21:02:27.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.882+0000 7fc1357e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.882+0000 7fc1357e5640 1 -- 172.21.15.106:0/2624920338 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1300ab5e0 con 0x7fc13006f400 2024-04-04T21:02:27.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.882+0000 7fc1357e5640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.882+0000 7fc1357e5640 10 monclient: _renew_subs 2024-04-04T21:02:27.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.882+0000 7fc1357e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.882+0000 7fc1357e5640 1 -- 172.21.15.106:0/2624920338 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1300abd80 con 0x7fc13006f400 2024-04-04T21:02:27.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.882+0000 7fc1357e5640 1 librados: init done 2024-04-04T21:02:27.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.884+0000 7fc1357e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.884+0000 7fc1357e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.884+0000 7fc1357e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:27.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.884+0000 7fc1357e5640 1 -- 172.21.15.106:0/2624920338 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13006a610 con 0x7fc13006f400 2024-04-04T21:02:27.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc311d9b640 1 -- 172.21.15.106:0/409546702 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc304034070 con 0x7fc31406ec20 2024-04-04T21:02:27.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc311d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc311d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:27.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc311d9b640 10 monclient: dump: 2024-04-04T21:02:27.886 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.886 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.886 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.886 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.887 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.887 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.887 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.887 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.887 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.887 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc311d9b640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc311d9b640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc311d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc311d9b640 1 -- 172.21.15.106:0/409546702 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc304030840 con 0x7fc31406ec20 2024-04-04T21:02:27.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc311d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc311d9b640 1 -- 172.21.15.106:0/409546702 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc304030c10 con 0x7fc31406ec20 2024-04-04T21:02:27.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc311d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc311d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:27.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc311d9b640 10 monclient: dump: 2024-04-04T21:02:27.888 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.888 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.888 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.888 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.888 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.888 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.888 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.888 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.888 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.888 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc318d68640 5 monclient: authenticate success, global_id 4329 2024-04-04T21:02:27.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc318d68640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:27.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.885+0000 7fc318d68640 4 set_mon_vals no callback set 2024-04-04T21:02:27.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.886+0000 7fc318d68640 10 monclient: shutdown 2024-04-04T21:02:27.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.886+0000 7fc318d68640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:27.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.886+0000 7fc318d68640 1 -- 172.21.15.106:0/409546702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc31406ec20 msgr2=0x7fc31413b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.886+0000 7fc318d68640 1 --2- 172.21.15.106:0/409546702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc31406ec20 0x7fc31413b550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc3040098a0 tx=0x7fc30402b980 comp rx=0 tx=0).stop 2024-04-04T21:02:27.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.886+0000 7fc318d68640 1 -- 172.21.15.106:0/409546702 shutdown_connections 2024-04-04T21:02:27.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.886+0000 7fc318d68640 1 --2- 172.21.15.106:0/409546702 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31413ba90 0x7fc3140715e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.886+0000 7fc318d68640 1 --2- 172.21.15.106:0/409546702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc31406ec20 0x7fc31413b550 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.886+0000 7fc318d68640 1 -- 172.21.15.106:0/409546702 >> 172.21.15.106:0/409546702 conn(0x7fc3140b3b90 msgr2=0x7fc3140b3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:27.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.886+0000 7fc318d68640 1 -- 172.21.15.106:0/409546702 shutdown_connections 2024-04-04T21:02:27.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.886+0000 7fc318d68640 1 -- 172.21.15.106:0/409546702 wait complete. 2024-04-04T21:02:27.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.888+0000 7fc318d68640 10 monclient: build_initial_monmap 2024-04-04T21:02:27.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.888+0000 7fc318d68640 10 monclient: monmap: 2024-04-04T21:02:27.892 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:27.892 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:27.892 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:27.892 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:27.892 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:27.892 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.892 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.888+0000 7fc318d68640 1 librados: starting msgr at 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.888+0000 7fc318d68640 1 librados: starting objecter 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.888+0000 7fc318d68640 1 Processor -- start 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.888+0000 7fc318d68640 1 -- start start 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.888+0000 7fc318d68640 1 librados: setting wanted keys 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.888+0000 7fc318d68640 1 librados: calling monclient init 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.888+0000 7fc318d68640 10 monclient: init 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc31406ec20 0x7fc314141930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 10 monclient(hunting): picked mon.noname-b con 0x7fc31406ec20 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:27.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc314071b20 0x7fc3140adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 10 monclient(hunting): picked mon.noname-a con 0x7fc314071b20 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:27.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31413ba90 0x7fc314132100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 10 monclient(hunting): picked mon.noname-c con 0x7fc31413ba90 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:27.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31413beb0 con 0x7fc314071b20 2024-04-04T21:02:27.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31406fd20 con 0x7fc31406ec20 2024-04-04T21:02:27.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:27.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc314072a60 con 0x7fc31413ba90 2024-04-04T21:02:27.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:27.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.889+0000 7fc318d68640 10 monclient(hunting): authenticate will time out at 1397.156383s 2024-04-04T21:02:27.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.890+0000 7fc312d9d640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31413ba90 0x7fc314132100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.890+0000 7fc312d9d640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31413ba90 0x7fc314132100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:32928/0 (socket says 172.21.15.106:32928) 2024-04-04T21:02:27.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.890+0000 7fc312d9d640 1 -- 172.21.15.106:0/2068065273 learned_addr learned my addr 172.21.15.106:0/2068065273 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:27.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.890+0000 7fc312d9d640 10 monclient(hunting): get_auth_request con 0x7fc31413ba90 auth_method 0 2024-04-04T21:02:27.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.890+0000 7fc312d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:27.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.890+0000 7fc312d9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:27.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.890+0000 7fc312d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:27.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.890+0000 7fc312d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:27.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.890+0000 7fc312d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:27.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.891+0000 7fc312d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:27.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.893+0000 7fc312d9d640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-04-04T21:02:27.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.893+0000 7fc312d9d640 1 -- 172.21.15.106:0/2068065273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc31406ec20 msgr2=0x7fc314141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.893+0000 7fc312d9d640 1 --2- 172.21.15.106:0/2068065273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc31406ec20 0x7fc314141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.893+0000 7fc312d9d640 1 -- 172.21.15.106:0/2068065273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc314071b20 msgr2=0x7fc3140adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:27.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.893+0000 7fc312d9d640 1 --2- 172.21.15.106:0/2068065273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc314071b20 0x7fc3140adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:27.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.893+0000 7fc312d9d640 10 monclient: _finish_hunting 0 2024-04-04T21:02:27.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.893+0000 7fc312d9d640 1 monclient: found mon.noname-c 2024-04-04T21:02:27.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.893+0000 7fc312d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:27.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.893+0000 7fc312d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.893+0000 7fc312d9d640 1 -- 172.21.15.106:0/2068065273 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc314141160 con 0x7fc31413ba90 2024-04-04T21:02:27.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.896+0000 7fc312d9d640 1 --2- 172.21.15.106:0/2068065273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31413ba90 0x7fc314132100 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00c780 tx=0x7fc30c00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.896+0000 7fc30affd640 10 client.?.objecter ms_handle_connect 0x7fc31413ba90 2024-04-04T21:02:27.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.896+0000 7fc30affd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:27.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc30affd640 1 -- 172.21.15.106:0/2068065273 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30c017070 con 0x7fc31413ba90 2024-04-04T21:02:27.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc30affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc30affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:27.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc30affd640 10 monclient: dump: 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc30affd640 10 monclient: _finish_auth 0 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc30affd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc30affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc30affd640 1 -- 172.21.15.106:0/2068065273 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30c003da0 con 0x7fc31413ba90 2024-04-04T21:02:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc30affd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc30affd640 1 -- 172.21.15.106:0/2068065273 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30c0052b0 con 0x7fc31413ba90 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc30affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc30affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc30affd640 10 monclient: dump: 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc318d68640 5 monclient: authenticate success, global_id 4307 2024-04-04T21:02:27.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc318d68640 10 monclient: _renew_subs 2024-04-04T21:02:27.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc31159a640 4 set_mon_vals no callback set 2024-04-04T21:02:27.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc318d68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc318d68640 1 -- 172.21.15.106:0/2068065273 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc31413e830 con 0x7fc31413ba90 2024-04-04T21:02:27.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc318d68640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:27.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc318d68640 10 monclient: _renew_subs 2024-04-04T21:02:27.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc318d68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc318d68640 1 -- 172.21.15.106:0/2068065273 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc314132820 con 0x7fc31413ba90 2024-04-04T21:02:27.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.900+0000 7fc318d68640 1 librados: init done 2024-04-04T21:02:27.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.902+0000 7fc318d68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.902+0000 7fc318d68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:27.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.902+0000 7fc318d68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:27.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.902+0000 7fc318d68640 1 -- 172.21.15.106:0/2068065273 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc31406e010 con 0x7fc31413ba90 2024-04-04T21:02:27.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.937+0000 7fc30affd640 1 -- 172.21.15.106:0/2068065273 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc30c006b10 con 0x7fc31413ba90 2024-04-04T21:02:27.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.938+0000 7fc30affd640 1 --2- 172.21.15.106:0/2068065273 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc2f406ce60 0x7fc2f406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.938+0000 7fc31259c640 1 --2- 172.21.15.106:0/2068065273 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc2f406ce60 0x7fc2f406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.938+0000 7fc31259c640 10 monclient: get_auth_request con 0x7fc2f406ce60 auth_method 0 2024-04-04T21:02:27.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.939+0000 7fc31259c640 1 --2- 172.21.15.106:0/2068065273 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc2f406ce60 0x7fc2f406f320 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc304036000 tx=0x7fc30403b040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:27.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.939+0000 7fc30affd640 10 client.4307.objecter ms_handle_connect 0x7fc2f406ce60 2024-04-04T21:02:27.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.947+0000 7fba236c2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-04T21:02:27.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.947+0000 7fba236c2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-04T21:02:27.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.947+0000 7fba236c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:27.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.947+0000 7fba236c2640 1 -- 172.21.15.106:0/3931018550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fba1c04f050 con 0x7fba1c0a9e30 2024-04-04T21:02:27.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.952+0000 7fc30affd640 1 -- 172.21.15.106:0/2068065273 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc30c006e00 con 0x7fc31413ba90 2024-04-04T21:02:27.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.952+0000 7fc30affd640 10 client.4307.objecter ms_dispatch 0x7fc314000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:27.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.952+0000 7fc30affd640 3 client.4307.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:27.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.952+0000 7fc30affd640 3 client.4307.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:27.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.952+0000 7fc30affd640 20 client.4307.objecter dump_active .. 0 homeless 2024-04-04T21:02:27.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.959+0000 7fc12cff9640 1 -- 172.21.15.106:0/2624920338 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc128041070 con 0x7fc13006f400 2024-04-04T21:02:27.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.959+0000 7fc12cff9640 1 --2- 172.21.15.106:0/2624920338 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc11c06cee0 0x7fc11c06f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:27.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.959+0000 7fc12ffff640 1 --2- 172.21.15.106:0/2624920338 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc11c06cee0 0x7fc11c06f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:27.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.959+0000 7fc12ffff640 10 monclient: get_auth_request con 0x7fc11c06cee0 auth_method 0 2024-04-04T21:02:27.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.960+0000 7fc12cff9640 10 client.4332.objecter ms_handle_connect 0x7fc11c06cee0 2024-04-04T21:02:27.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:27.960+0000 7fc12ffff640 1 --2- 172.21.15.106:0/2624920338 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc11c06cee0 0x7fc11c06f3a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc124006a90 tx=0x7fc124006a20 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:28.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.030+0000 7f2af1271640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-04T21:02:28.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.030+0000 7f2af1271640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-04T21:02:28.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.030+0000 7f2af1271640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:28.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.030+0000 7f2af1271640 1 -- 172.21.15.106:0/3240996473 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f2ae4005e40 con 0x7f2ae4006df0 2024-04-04T21:02:28.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.069+0000 7fdf22a47640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-04-04T21:02:28.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.069+0000 7fdf22a47640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-04-04T21:02:28.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.069+0000 7fdf22a47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:28.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.069+0000 7fdf22a47640 1 -- 172.21.15.106:0/1831757200 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fdf1c04f050 con 0x7fdf1c147730 2024-04-04T21:02:28.192 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.192+0000 7fc30affd640 1 -- 172.21.15.106:0/2068065273 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc31406e010 con 0x7fc31413ba90 2024-04-04T21:02:28.193 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.192+0000 7fc30affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:28.193 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.192+0000 7fc30affd640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:28.200 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.200+0000 7f2ad37fe640 1 -- 172.21.15.106:0/3240996473 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2ae00093b0 con 0x7f2ae4006df0 2024-04-04T21:02:28.200 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.200+0000 7f2ad37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-04T21:02:28.201 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.200+0000 7f2ad37fe640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:28.201 INFO:teuthology.orchestra.run.smithi106.stdout:64424509445 2024-04-04T21:02:28.203 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.202+0000 7f2af1271640 10 librados: watch_flush enter 2024-04-04T21:02:28.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.202+0000 7f2af1271640 10 librados: watch_flush exit 2024-04-04T21:02:28.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.202+0000 7f2af1271640 20 client.4298.objecter shutdown clearing up homeless session... 2024-04-04T21:02:28.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.202+0000 7f2af1271640 10 client.4298.objecter successfully canceled tick 2024-04-04T21:02:28.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.202+0000 7f2af1271640 1 -- 172.21.15.106:0/3240996473 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2ad806ce60 msgr2=0x7f2ad806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.203+0000 7f2af1271640 1 --2- 172.21.15.106:0/3240996473 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2ad806ce60 0x7f2ad806f320 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2ae40c9ed0 tx=0x7f2adc009060 comp rx=0 tx=0).stop 2024-04-04T21:02:28.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.203+0000 7f2af1271640 10 monclient: shutdown 2024-04-04T21:02:28.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.203+0000 7f2af1271640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:28.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.203+0000 7f2af1271640 1 -- 172.21.15.106:0/3240996473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ae4006df0 msgr2=0x7f2ae40d0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.203+0000 7f2af1271640 1 --2- 172.21.15.106:0/3240996473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ae4006df0 0x7f2ae40d0af0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0009890 tx=0x7f2ae00046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.203+0000 7f2af1271640 1 -- 172.21.15.106:0/3240996473 shutdown_connections 2024-04-04T21:02:28.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.203+0000 7f2af1271640 1 --2- 172.21.15.106:0/3240996473 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2ad806ce60 0x7f2ad806f320 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.203+0000 7f2af1271640 1 --2- 172.21.15.106:0/3240996473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ae4030e40 0x7f2ae40e6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.203+0000 7f2af1271640 1 --2- 172.21.15.106:0/3240996473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae402f980 0x7f2ae40ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.203+0000 7f2af1271640 1 --2- 172.21.15.106:0/3240996473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ae4006df0 0x7f2ae40d0af0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.203+0000 7f2af1271640 1 -- 172.21.15.106:0/3240996473 >> 172.21.15.106:0/3240996473 conn(0x7f2ae4006060 msgr2=0x7f2ae402e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:28.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.203+0000 7f2af1271640 1 -- 172.21.15.106:0/3240996473 shutdown_connections 2024-04-04T21:02:28.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.203+0000 7f2af1271640 1 -- 172.21.15.106:0/3240996473 wait complete. 2024-04-04T21:02:28.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.203+0000 7f2af1271640 1 librados: shutdown 2024-04-04T21:02:28.218 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.0 2024-04-04T21:02:28.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.233+0000 7fc12cff9640 1 -- 172.21.15.106:0/2624920338 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc12807d930 con 0x7fc13006f400 2024-04-04T21:02:28.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.233+0000 7fc12cff9640 10 client.4332.objecter ms_dispatch 0x7fc130000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:28.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.233+0000 7fc12cff9640 3 client.4332.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:28.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.233+0000 7fc12cff9640 3 client.4332.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:28.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.233+0000 7fc12cff9640 20 client.4332.objecter dump_active .. 0 homeless 2024-04-04T21:02:28.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.233+0000 7fc12cff9640 1 -- 172.21.15.106:0/2624920338 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc128051070 con 0x7fc13006f400 2024-04-04T21:02:28.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.233+0000 7fc12cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:28.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.233+0000 7fc12cff9640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:28.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.254+0000 7fdf18ff9640 1 -- 172.21.15.106:0/1831757200 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdf10058cd0 con 0x7fdf1c147730 2024-04-04T21:02:28.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.254+0000 7fdf18ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-04-04T21:02:28.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.254+0000 7fdf18ff9640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:28.255 INFO:teuthology.orchestra.run.smithi106.stdout:64424509447 2024-04-04T21:02:28.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 10 librados: watch_flush enter 2024-04-04T21:02:28.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 10 librados: watch_flush exit 2024-04-04T21:02:28.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 20 client.4326.objecter shutdown clearing up homeless session... 2024-04-04T21:02:28.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 10 client.4326.objecter successfully canceled tick 2024-04-04T21:02:28.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 1 -- 172.21.15.106:0/1831757200 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdf0806ceb0 msgr2=0x7fdf0806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 1 --2- 172.21.15.106:0/1831757200 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdf0806ceb0 0x7fdf0806f370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c006a90 tx=0x7fdf0c006a20 comp rx=0 tx=0).stop 2024-04-04T21:02:28.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 10 monclient: shutdown 2024-04-04T21:02:28.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:28.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 1 -- 172.21.15.106:0/1831757200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf1c147730 msgr2=0x7fdf1c124fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 1 --2- 172.21.15.106:0/1831757200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf1c147730 0x7fdf1c124fd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdf1000c3f0 tx=0x7fdf1000c8b0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 1 -- 172.21.15.106:0/1831757200 shutdown_connections 2024-04-04T21:02:28.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 1 --2- 172.21.15.106:0/1831757200 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdf0806ceb0 0x7fdf0806f370 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 1 --2- 172.21.15.106:0/1831757200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf1c147730 0x7fdf1c124fd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 1 --2- 172.21.15.106:0/1831757200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf1c06fa70 0x7fdf1c12a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 1 --2- 172.21.15.106:0/1831757200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf1c06e810 0x7fdf1c13e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 1 -- 172.21.15.106:0/1831757200 >> 172.21.15.106:0/1831757200 conn(0x7fdf1c0b3b90 msgr2=0x7fdf1c0b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:28.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.256+0000 7fdf22a47640 1 -- 172.21.15.106:0/1831757200 shutdown_connections 2024-04-04T21:02:28.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.257+0000 7fdf22a47640 1 -- 172.21.15.106:0/1831757200 wait complete. 2024-04-04T21:02:28.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.257+0000 7fdf22a47640 1 librados: shutdown 2024-04-04T21:02:28.269 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509447 for osd.6 2024-04-04T21:02:28.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.336+0000 7f56777fe640 1 -- 172.21.15.106:0/1442672833 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f568404e470 con 0x7f56900a9e30 2024-04-04T21:02:28.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.336+0000 7f56777fe640 1 --2- 172.21.15.106:0/1442672833 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f567c06ceb0 0x7f567c06f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:28.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.336+0000 7f568ed9d640 1 --2- 172.21.15.106:0/1442672833 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f567c06ceb0 0x7f567c06f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:28.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.336+0000 7f568ed9d640 10 monclient: get_auth_request con 0x7f567c06ceb0 auth_method 0 2024-04-04T21:02:28.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.336+0000 7f56777fe640 10 client.4366.objecter ms_handle_connect 0x7f567c06ceb0 2024-04-04T21:02:28.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.336+0000 7f568ed9d640 1 --2- 172.21.15.106:0/1442672833 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f567c06ceb0 0x7f567c06f370 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5680001560 tx=0x7f5680007990 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:28.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.341+0000 7f56777fe640 1 -- 172.21.15.106:0/1442672833 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f568402f030 con 0x7f56900a9e30 2024-04-04T21:02:28.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.341+0000 7f56777fe640 10 client.4366.objecter ms_dispatch 0x7f5690000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:28.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.341+0000 7f56777fe640 3 client.4366.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:28.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.341+0000 7f56777fe640 3 client.4366.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:28.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.342+0000 7f56777fe640 20 client.4366.objecter dump_active .. 0 homeless 2024-04-04T21:02:28.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f286e7fc640 1 -- 172.21.15.106:0/1279842228 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874032880 con 0x7f28780a9b60 2024-04-04T21:02:28.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f286e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:28.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f286e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:28.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f286e7fc640 10 monclient: dump: 2024-04-04T21:02:28.355 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:28.355 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:28.355 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:28.355 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:28.355 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f286e7fc640 10 monclient: _finish_auth 0 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f286e7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f286e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f286e7fc640 1 -- 172.21.15.106:0/1279842228 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2874032a40 con 0x7f28780a9b60 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f286e7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f286e7fc640 1 -- 172.21.15.106:0/1279842228 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874032e30 con 0x7f28780a9b60 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f287fae1640 5 monclient: authenticate success, global_id 4369 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f287fae1640 10 monclient: _renew_subs 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f287fae1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:28.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f287fae1640 1 -- 172.21.15.106:0/1279842228 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287813af20 con 0x7f28780a9b60 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f286ffff640 4 set_mon_vals no callback set 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f286e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f286e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f286e7fc640 10 monclient: dump: 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:28.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f287fae1640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:28.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f287fae1640 10 monclient: _renew_subs 2024-04-04T21:02:28.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f287fae1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:28.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f287fae1640 1 -- 172.21.15.106:0/1279842228 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2878139de0 con 0x7f28780a9b60 2024-04-04T21:02:28.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7f287fae1640 1 librados: init done 2024-04-04T21:02:28.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.354+0000 7fc25effd640 1 -- 172.21.15.106:0/189408947 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc26001a070 con 0x7fc268142880 2024-04-04T21:02:28.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.355+0000 7fc25effd640 1 --2- 172.21.15.106:0/189408947 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc24c06cee0 0x7fc24c06f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:28.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.355+0000 7fc26e272640 1 --2- 172.21.15.106:0/189408947 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc24c06cee0 0x7fc24c06f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:28.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.355+0000 7fc26e272640 10 monclient: get_auth_request con 0x7fc24c06cee0 auth_method 0 2024-04-04T21:02:28.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.355+0000 7f287fae1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:28.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.355+0000 7f287fae1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:28.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.355+0000 7f287fae1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:28.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.355+0000 7f287fae1640 1 -- 172.21.15.106:0/1279842228 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2878140360 con 0x7f28780a9b60 2024-04-04T21:02:28.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.355+0000 7fc25effd640 10 client.4363.objecter ms_handle_connect 0x7fc24c06cee0 2024-04-04T21:02:28.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.355+0000 7fc26e272640 1 --2- 172.21.15.106:0/189408947 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc24c06cee0 0x7fc24c06f3a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc264036000 tx=0x7fc264030000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:28.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.357+0000 7fc25effd640 1 -- 172.21.15.106:0/189408947 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc260056b40 con 0x7fc268142880 2024-04-04T21:02:28.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.357+0000 7fc25effd640 10 client.4363.objecter ms_dispatch 0x7fc268000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:28.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.357+0000 7fc25effd640 3 client.4363.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:28.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.357+0000 7fc25effd640 3 client.4363.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:28.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.357+0000 7fc25effd640 20 client.4363.objecter dump_active .. 0 homeless 2024-04-04T21:02:28.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.369+0000 7f56777fe640 1 -- 172.21.15.106:0/1442672833 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5684040070 con 0x7f56900a9e30 2024-04-04T21:02:28.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.369+0000 7f56777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:28.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.369+0000 7f56777fe640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:28.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.380+0000 7fc318d68640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-04T21:02:28.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.380+0000 7fc318d68640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-04T21:02:28.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.380+0000 7fc318d68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:28.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.380+0000 7fc318d68640 1 -- 172.21.15.106:0/2068065273 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc31404f050 con 0x7fc31413ba90 2024-04-04T21:02:28.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.385+0000 7fc30affd640 1 -- 172.21.15.106:0/2068065273 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc30c020080 con 0x7fc31413ba90 2024-04-04T21:02:28.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.385+0000 7fc30affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-04T21:02:28.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.385+0000 7fc30affd640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:28.385 INFO:teuthology.orchestra.run.smithi106.stdout:85899345927 2024-04-04T21:02:28.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.386+0000 7fc25effd640 1 -- 172.21.15.106:0/189408947 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc260021020 con 0x7fc268142880 2024-04-04T21:02:28.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.386+0000 7fc25effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:28.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.386+0000 7fc25effd640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:28.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.386+0000 7fc318d68640 10 librados: watch_flush enter 2024-04-04T21:02:28.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.386+0000 7fc318d68640 10 librados: watch_flush exit 2024-04-04T21:02:28.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.386+0000 7fc318d68640 20 client.4307.objecter shutdown clearing up homeless session... 2024-04-04T21:02:28.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.386+0000 7fc318d68640 10 client.4307.objecter successfully canceled tick 2024-04-04T21:02:28.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 1 -- 172.21.15.106:0/2068065273 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc2f406ce60 msgr2=0x7fc2f406f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 1 --2- 172.21.15.106:0/2068065273 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc2f406ce60 0x7fc2f406f320 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc304036000 tx=0x7fc30403b040 comp rx=0 tx=0).stop 2024-04-04T21:02:28.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 10 monclient: shutdown 2024-04-04T21:02:28.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:28.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 1 -- 172.21.15.106:0/2068065273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31413ba90 msgr2=0x7fc314132100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 1 --2- 172.21.15.106:0/2068065273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31413ba90 0x7fc314132100 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00c780 tx=0x7fc30c00cc40 comp rx=0 tx=0).stop 2024-04-04T21:02:28.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 1 -- 172.21.15.106:0/2068065273 shutdown_connections 2024-04-04T21:02:28.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 1 --2- 172.21.15.106:0/2068065273 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc2f406ce60 0x7fc2f406f320 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 1 --2- 172.21.15.106:0/2068065273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31413ba90 0x7fc314132100 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.393 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 1 --2- 172.21.15.106:0/2068065273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc314071b20 0x7fc3140adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.393 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 1 --2- 172.21.15.106:0/2068065273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc31406ec20 0x7fc314141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.393 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 1 -- 172.21.15.106:0/2068065273 >> 172.21.15.106:0/2068065273 conn(0x7fc3140b3b90 msgr2=0x7fc314139970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:28.393 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 1 -- 172.21.15.106:0/2068065273 shutdown_connections 2024-04-04T21:02:28.393 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 1 -- 172.21.15.106:0/2068065273 wait complete. 2024-04-04T21:02:28.393 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.387+0000 7fc318d68640 1 librados: shutdown 2024-04-04T21:02:28.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.397+0000 7fba11ffb640 1 -- 172.21.15.106:0/3931018550 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fba0c07ac70 con 0x7fba1c0a9e30 2024-04-04T21:02:28.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.397+0000 7fba11ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-04T21:02:28.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.397+0000 7fba11ffb640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:28.398 INFO:teuthology.orchestra.run.smithi106.stdout:55834574853 2024-04-04T21:02:28.400 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345928 got 85899345927 for osd.3 2024-04-04T21:02:28.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 10 librados: watch_flush enter 2024-04-04T21:02:28.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 10 librados: watch_flush exit 2024-04-04T21:02:28.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 20 client.4345.objecter shutdown clearing up homeless session... 2024-04-04T21:02:28.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 10 client.4345.objecter successfully canceled tick 2024-04-04T21:02:28.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 1 -- 172.21.15.106:0/3931018550 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fba0006d290 msgr2=0x7fba0006f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 1 --2- 172.21.15.106:0/3931018550 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fba0006d290 0x7fba0006f750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fba18005880 tx=0x7fba18009040 comp rx=0 tx=0).stop 2024-04-04T21:02:28.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 10 monclient: shutdown 2024-04-04T21:02:28.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:28.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 1 -- 172.21.15.106:0/3931018550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0a9e30 msgr2=0x7fba1c1d7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 1 --2- 172.21.15.106:0/3931018550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0a9e30 0x7fba1c1d7b60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fba0c034040 tx=0x7fba0c0046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 1 -- 172.21.15.106:0/3931018550 shutdown_connections 2024-04-04T21:02:28.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 1 --2- 172.21.15.106:0/3931018550 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fba0006d290 0x7fba0006f750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 1 --2- 172.21.15.106:0/3931018550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba1c142dc0 0x7fba1c1d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 1 --2- 172.21.15.106:0/3931018550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba1c0a9e30 0x7fba1c1d7b60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 1 --2- 172.21.15.106:0/3931018550 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c0a9a50 0x7fba1c143bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.399+0000 7fba236c2640 1 -- 172.21.15.106:0/3931018550 >> 172.21.15.106:0/3931018550 conn(0x7fba1c0b8030 msgr2=0x7fba1c0b8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:28.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.400+0000 7fba236c2640 1 -- 172.21.15.106:0/3931018550 shutdown_connections 2024-04-04T21:02:28.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.400+0000 7fba236c2640 1 -- 172.21.15.106:0/3931018550 wait complete. 2024-04-04T21:02:28.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.400+0000 7fba236c2640 1 librados: shutdown 2024-04-04T21:02:28.411 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-04-04T21:02:28.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.422+0000 7fc1357e5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-04T21:02:28.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.422+0000 7fc1357e5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-04T21:02:28.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.422+0000 7fc1357e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:28.423 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.422+0000 7fc1357e5640 1 -- 172.21.15.106:0/2624920338 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc1300ab7b0 con 0x7fc13006f400 2024-04-04T21:02:28.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.425+0000 7fc12cff9640 1 -- 172.21.15.106:0/2624920338 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc12807ff90 con 0x7fc13006f400 2024-04-04T21:02:28.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.425+0000 7fc12cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-04T21:02:28.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.425+0000 7fc12cff9640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:28.426 INFO:teuthology.orchestra.run.smithi106.stdout:85899345928 2024-04-04T21:02:28.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 10 librados: watch_flush enter 2024-04-04T21:02:28.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 10 librados: watch_flush exit 2024-04-04T21:02:28.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 20 client.4332.objecter shutdown clearing up homeless session... 2024-04-04T21:02:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 10 client.4332.objecter successfully canceled tick 2024-04-04T21:02:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 1 -- 172.21.15.106:0/2624920338 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc11c06cee0 msgr2=0x7fc11c06f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 1 --2- 172.21.15.106:0/2624920338 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc11c06cee0 0x7fc11c06f3a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc124006a90 tx=0x7fc124006a20 comp rx=0 tx=0).stop 2024-04-04T21:02:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 10 monclient: shutdown 2024-04-04T21:02:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 1 -- 172.21.15.106:0/2624920338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13006f400 msgr2=0x7fc130136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 1 --2- 172.21.15.106:0/2624920338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13006f400 0x7fc130136d80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc128008b40 tx=0x7fc128004a40 comp rx=0 tx=0).stop 2024-04-04T21:02:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 1 -- 172.21.15.106:0/2624920338 shutdown_connections 2024-04-04T21:02:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 1 --2- 172.21.15.106:0/2624920338 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc11c06cee0 0x7fc11c06f3a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 1 --2- 172.21.15.106:0/2624920338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1300ab100 0x7fc1301372c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 1 --2- 172.21.15.106:0/2624920338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc13006f400 0x7fc130136d80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 1 --2- 172.21.15.106:0/2624920338 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc13006f020 0x7fc13013e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.427+0000 7fc1357e5640 1 -- 172.21.15.106:0/2624920338 >> 172.21.15.106:0/2624920338 conn(0x7fc1300b3c30 msgr2=0x7fc13013b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.428+0000 7fc1357e5640 1 -- 172.21.15.106:0/2624920338 shutdown_connections 2024-04-04T21:02:28.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.428+0000 7fc1357e5640 1 -- 172.21.15.106:0/2624920338 wait complete. 2024-04-04T21:02:28.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.428+0000 7fc1357e5640 1 librados: shutdown 2024-04-04T21:02:28.440 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345928 for osd.1 2024-04-04T21:02:28.596 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.596+0000 7f5695005640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-04T21:02:28.596 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.596+0000 7f5695005640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-04T21:02:28.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.596+0000 7f5695005640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:28.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.596+0000 7f5695005640 1 -- 172.21.15.106:0/1442672833 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f56900b7b90 con 0x7f56900a9e30 2024-04-04T21:02:28.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.611+0000 7fc2704d6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-04T21:02:28.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.611+0000 7fc2704d6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-04T21:02:28.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.611+0000 7fc2704d6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:28.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.611+0000 7fc2704d6640 1 -- 172.21.15.106:0/189408947 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fc2680b7b30 con 0x7fc268142880 2024-04-04T21:02:28.658 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.658+0000 7f286e7fc640 1 -- 172.21.15.106:0/1279842228 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f287404c060 con 0x7f28780a9b60 2024-04-04T21:02:28.659 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.658+0000 7f286e7fc640 1 --2- 172.21.15.106:0/1279842228 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f285c06ceb0 0x7f285c06f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:28.659 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.658+0000 7f287d07c640 1 --2- 172.21.15.106:0/1279842228 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f285c06ceb0 0x7f285c06f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:28.659 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.658+0000 7f287d07c640 10 monclient: get_auth_request con 0x7f285c06ceb0 auth_method 0 2024-04-04T21:02:28.659 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.659+0000 7f286e7fc640 10 client.4369.objecter ms_handle_connect 0x7f285c06ceb0 2024-04-04T21:02:28.659 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.659+0000 7f287d07c640 1 --2- 172.21.15.106:0/1279842228 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f285c06ceb0 0x7f285c06f370 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f28781358f0 tx=0x7f2868006ba0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:28.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.661+0000 7f286e7fc640 1 -- 172.21.15.106:0/1279842228 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2874078600 con 0x7f28780a9b60 2024-04-04T21:02:28.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.661+0000 7f286e7fc640 10 client.4369.objecter ms_dispatch 0x7f2878000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:28.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.661+0000 7f286e7fc640 3 client.4369.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:28.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.661+0000 7f286e7fc640 3 client.4369.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:28.662 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.661+0000 7f286e7fc640 20 client.4369.objecter dump_active .. 0 homeless 2024-04-04T21:02:28.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.670+0000 7f286e7fc640 1 -- 172.21.15.106:0/1279842228 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2874040070 con 0x7f28780a9b60 2024-04-04T21:02:28.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.670+0000 7f286e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:28.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.670+0000 7f286e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:28.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.678+0000 7fc25effd640 1 -- 172.21.15.106:0/189408947 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc268143bb0 con 0x7fc268142880 2024-04-04T21:02:28.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.678+0000 7fc25effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-04T21:02:28.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.678+0000 7fc25effd640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:28.678 INFO:teuthology.orchestra.run.smithi106.stdout:60129542150 2024-04-04T21:02:28.679 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.679+0000 7f56777fe640 1 -- 172.21.15.106:0/1442672833 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f568407b1f0 con 0x7f56900a9e30 2024-04-04T21:02:28.679 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.679+0000 7f56777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-04T21:02:28.679 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.679+0000 7f56777fe640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:28.680 INFO:teuthology.orchestra.run.smithi106.stdout:85899345928 2024-04-04T21:02:28.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.679+0000 7fc2704d6640 10 librados: watch_flush enter 2024-04-04T21:02:28.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.679+0000 7fc2704d6640 10 librados: watch_flush exit 2024-04-04T21:02:28.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.679+0000 7fc2704d6640 20 client.4363.objecter shutdown clearing up homeless session... 2024-04-04T21:02:28.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.679+0000 7fc2704d6640 10 client.4363.objecter successfully canceled tick 2024-04-04T21:02:28.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 1 -- 172.21.15.106:0/189408947 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc24c06cee0 msgr2=0x7fc24c06f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 1 --2- 172.21.15.106:0/189408947 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc24c06cee0 0x7fc24c06f3a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc264036000 tx=0x7fc264030000 comp rx=0 tx=0).stop 2024-04-04T21:02:28.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 10 monclient: shutdown 2024-04-04T21:02:28.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:28.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 1 -- 172.21.15.106:0/189408947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc268142880 msgr2=0x7fc2681dbca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 1 --2- 172.21.15.106:0/189408947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc268142880 0x7fc2681dbca0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc26000ca20 tx=0x7fc26000cee0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 1 -- 172.21.15.106:0/189408947 shutdown_connections 2024-04-04T21:02:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 1 --2- 172.21.15.106:0/189408947 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc24c06cee0 0x7fc24c06f3a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 1 --2- 172.21.15.106:0/189408947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc268142880 0x7fc2681dbca0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 1 --2- 172.21.15.106:0/189408947 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc26813f6a0 0x7fc268135d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 1 --2- 172.21.15.106:0/189408947 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2680a9a50 0x7fc2681db180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 1 -- 172.21.15.106:0/189408947 >> 172.21.15.106:0/189408947 conn(0x7fc2680b8030 msgr2=0x7fc268142d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 1 -- 172.21.15.106:0/189408947 shutdown_connections 2024-04-04T21:02:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 1 -- 172.21.15.106:0/189408947 wait complete. 2024-04-04T21:02:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.680+0000 7fc2704d6640 1 librados: shutdown 2024-04-04T21:02:28.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 10 librados: watch_flush enter 2024-04-04T21:02:28.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 10 librados: watch_flush exit 2024-04-04T21:02:28.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 20 client.4366.objecter shutdown clearing up homeless session... 2024-04-04T21:02:28.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 10 client.4366.objecter successfully canceled tick 2024-04-04T21:02:28.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 1 -- 172.21.15.106:0/1442672833 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f567c06ceb0 msgr2=0x7f567c06f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 1 --2- 172.21.15.106:0/1442672833 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f567c06ceb0 0x7f567c06f370 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5680001560 tx=0x7f5680007990 comp rx=0 tx=0).stop 2024-04-04T21:02:28.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 10 monclient: shutdown 2024-04-04T21:02:28.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:28.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 1 -- 172.21.15.106:0/1442672833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56900a9e30 msgr2=0x7f5690143960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 1 --2- 172.21.15.106:0/1442672833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56900a9e30 0x7f5690143960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f56840098c0 tx=0x7f56840046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 1 -- 172.21.15.106:0/1442672833 shutdown_connections 2024-04-04T21:02:28.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 1 --2- 172.21.15.106:0/1442672833 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f567c06ceb0 0x7f567c06f370 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 1 --2- 172.21.15.106:0/1442672833 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5690142dc0 0x7f5690143ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 1 --2- 172.21.15.106:0/1442672833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56900a9e30 0x7f5690143960 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 1 --2- 172.21.15.106:0/1442672833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56900a9a50 0x7f5690137cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 1 -- 172.21.15.106:0/1442672833 >> 172.21.15.106:0/1442672833 conn(0x7f56900b8030 msgr2=0x7f56900b8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:28.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 1 -- 172.21.15.106:0/1442672833 shutdown_connections 2024-04-04T21:02:28.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 1 -- 172.21.15.106:0/1442672833 wait complete. 2024-04-04T21:02:28.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.682+0000 7f5695005640 1 librados: shutdown 2024-04-04T21:02:28.692 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.7 2024-04-04T21:02:28.698 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345928 for osd.2 2024-04-04T21:02:28.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.887+0000 7f287fae1640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-04T21:02:28.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.887+0000 7f287fae1640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-04T21:02:28.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.887+0000 7f287fae1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:28.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.887+0000 7f287fae1640 1 -- 172.21.15.106:0/1279842228 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f287804f050 con 0x7f28780a9b60 2024-04-04T21:02:28.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.892+0000 7f286e7fc640 1 -- 172.21.15.106:0/1279842228 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f287404b2c0 con 0x7f28780a9b60 2024-04-04T21:02:28.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.892+0000 7f286e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-04T21:02:28.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.892+0000 7f286e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:28.892 INFO:teuthology.orchestra.run.smithi106.stdout:77309411337 2024-04-04T21:02:28.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.893+0000 7f287fae1640 10 librados: watch_flush enter 2024-04-04T21:02:28.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.893+0000 7f287fae1640 10 librados: watch_flush exit 2024-04-04T21:02:28.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.893+0000 7f287fae1640 20 client.4369.objecter shutdown clearing up homeless session... 2024-04-04T21:02:28.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.893+0000 7f287fae1640 10 client.4369.objecter successfully canceled tick 2024-04-04T21:02:28.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.893+0000 7f287fae1640 1 -- 172.21.15.106:0/1279842228 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f285c06ceb0 msgr2=0x7f285c06f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.893+0000 7f287fae1640 1 --2- 172.21.15.106:0/1279842228 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f285c06ceb0 0x7f285c06f370 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f28781358f0 tx=0x7f2868006ba0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.893+0000 7f287fae1640 10 monclient: shutdown 2024-04-04T21:02:28.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.893+0000 7f287fae1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:28.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.893+0000 7f287fae1640 1 -- 172.21.15.106:0/1279842228 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28780a9b60 msgr2=0x7f2878138e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:28.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.893+0000 7f287fae1640 1 --2- 172.21.15.106:0/1279842228 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28780a9b60 0x7f2878138e10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2874000c00 tx=0x7f28740046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.893+0000 7f287fae1640 1 -- 172.21.15.106:0/1279842228 shutdown_connections 2024-04-04T21:02:28.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.893+0000 7f287fae1640 1 --2- 172.21.15.106:0/1279842228 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f285c06ceb0 0x7f285c06f370 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.893+0000 7f287fae1640 1 --2- 172.21.15.106:0/1279842228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2878137290 0x7f28781398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.893+0000 7f287fae1640 1 --2- 172.21.15.106:0/1279842228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f287813f810 0x7f2878134ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.894+0000 7f287fae1640 1 --2- 172.21.15.106:0/1279842228 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28780a9b60 0x7f2878138e10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:28.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.894+0000 7f287fae1640 1 -- 172.21.15.106:0/1279842228 >> 172.21.15.106:0/1279842228 conn(0x7f28780b7ec0 msgr2=0x7f287813b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:28.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.894+0000 7f287fae1640 1 -- 172.21.15.106:0/1279842228 shutdown_connections 2024-04-04T21:02:28.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.894+0000 7f287fae1640 1 -- 172.21.15.106:0/1279842228 wait complete. 2024-04-04T21:02:28.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:28.894+0000 7f287fae1640 1 librados: shutdown 2024-04-04T21:02:28.905 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411338 got 77309411337 for osd.4 2024-04-04T21:02:29.218 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-04T21:02:29.270 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-04T21:02:29.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.322+0000 7fe69c33d640 10 monclient: get_monmap_and_config 2024-04-04T21:02:29.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.322+0000 7fe69c33d640 10 monclient: build_initial_monmap 2024-04-04T21:02:29.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.322+0000 7fe69c33d640 10 monclient: monmap: 2024-04-04T21:02:29.323 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:29.323 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.323 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:29.323576+0000 2024-04-04T21:02:29.323 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:29.323576+0000 2024-04-04T21:02:29.324 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:29.324 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.324 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:29.324 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:29.324 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:29.324 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:29.324 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:29.324 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 1 Processor -- start 2024-04-04T21:02:29.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 1 -- start start 2024-04-04T21:02:29.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 10 monclient: init 2024-04-04T21:02:29.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-04-04T21:02:29.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6940a9a50 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:29.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6940a9e30 0x7fe694142880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 10 monclient(hunting): picked mon.noname-b con 0x7fe6940a9e30 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:29.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe694142dc0 0x7fe69414cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 10 monclient(hunting): picked mon.noname-c con 0x7fe694142dc0 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:29.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6940ab750 con 0x7fe6940a9a50 2024-04-04T21:02:29.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe694140120 con 0x7fe694142dc0 2024-04-04T21:02:29.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6940aa660 con 0x7fe6940a9e30 2024-04-04T21:02:29.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.323+0000 7fe69c33d640 10 monclient(hunting): authenticate will time out at 1398.591153s 2024-04-04T21:02:29.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.324+0000 7fe6998d8640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6940a9e30 0x7fe694142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.324+0000 7fe69a8da640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe694142dc0 0x7fe69414cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.324+0000 7fe69a8da640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe694142dc0 0x7fe69414cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50342/0 (socket says 172.21.15.106:50342) 2024-04-04T21:02:29.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.324+0000 7fe69a8da640 1 -- 172.21.15.106:0/454494631 learned_addr learned my addr 172.21.15.106:0/454494631 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:29.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.324+0000 7fe69a8da640 10 monclient(hunting): get_auth_request con 0x7fe694142dc0 auth_method 0 2024-04-04T21:02:29.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.324+0000 7fe69a8da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.324+0000 7fe69a8da640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.324+0000 7fe69a8da640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.324+0000 7fe6998d8640 10 monclient(hunting): get_auth_request con 0x7fe6940a9e30 auth_method 0 2024-04-04T21:02:29.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.324+0000 7fe6998d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.324+0000 7fe6998d8640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.324+0000 7fe6998d8640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.325+0000 7fe6998d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.325+0000 7fe6998d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.325+0000 7fe6998d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:29.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.325+0000 7fe69a8da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.325+0000 7fe69a8da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.325+0000 7fe69a8da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:29.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.327+0000 7fe6998d8640 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2024-04-04T21:02:29.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.327+0000 7fe6998d8640 1 -- 172.21.15.106:0/454494631 >> v2:172.21.15.203:3300/0 conn(0x7fe694142dc0 msgr2=0x7fe69414cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:29.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.327+0000 7fe6998d8640 1 --2- 172.21.15.106:0/454494631 >> v2:172.21.15.203:3300/0 conn(0x7fe694142dc0 0x7fe69414cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.327+0000 7fe6998d8640 1 -- 172.21.15.106:0/454494631 >> v1:172.21.15.203:6789/0 conn(0x7fe6940a9a50 legacy=0x7fe69413f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.327+0000 7fe6998d8640 10 monclient: _finish_hunting 0 2024-04-04T21:02:29.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.327+0000 7fe6998d8640 1 monclient: found mon.noname-b 2024-04-04T21:02:29.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.327+0000 7fe6998d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:29.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.327+0000 7fe6998d8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:29.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.327+0000 7fe6998d8640 1 -- 172.21.15.106:0/454494631 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6940aa250 con 0x7fe6940a9e30 2024-04-04T21:02:29.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.329+0000 7fe6998d8640 1 --2- 172.21.15.106:0/454494631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6940a9e30 0x7fe694142880 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe6840097a0 tx=0x7fe68402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81227c02f96f7039 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe6990d7640 1 -- 172.21.15.106:0/454494631 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6840315a0 con 0x7fe6940a9e30 2024-04-04T21:02:29.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe6990d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe6990d7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:29.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe6990d7640 10 monclient: dump: 2024-04-04T21:02:29.335 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.336 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.336 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.336 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.336 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.336 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.336 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.336 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.336 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.336 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe6990d7640 10 monclient: _finish_auth 0 2024-04-04T21:02:29.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe6990d7640 10 monclient: _check_auth_tickets 2024-04-04T21:02:29.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe6990d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:29.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe69c33d640 5 monclient: authenticate success, global_id 4310 2024-04-04T21:02:29.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe6990d7640 1 -- 172.21.15.106:0/454494631 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe684037070 con 0x7fe6940a9e30 2024-04-04T21:02:29.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe69c33d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:29.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe6990d7640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:29.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe69c33d640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:29.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe69c33d640 4 set_mon_vals no callback set 2024-04-04T21:02:29.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe6990d7640 1 -- 172.21.15.106:0/454494631 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe684031960 con 0x7fe6940a9e30 2024-04-04T21:02:29.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe6990d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe6990d7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:29.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe6990d7640 10 monclient: dump: 2024-04-04T21:02:29.337 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.337 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.337 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.337 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.338 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.338 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.338 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.338 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.338 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.338 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe69c33d640 10 monclient: shutdown 2024-04-04T21:02:29.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe69c33d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:29.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe69c33d640 1 -- 172.21.15.106:0/454494631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6940a9e30 msgr2=0x7fe694142880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.333+0000 7fe69c33d640 1 --2- 172.21.15.106:0/454494631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6940a9e30 0x7fe694142880 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe6840097a0 tx=0x7fe68402b840 comp rx=0 tx=0).stop 2024-04-04T21:02:29.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.334+0000 7fe69c33d640 1 -- 172.21.15.106:0/454494631 shutdown_connections 2024-04-04T21:02:29.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.334+0000 7fe69c33d640 1 --2- 172.21.15.106:0/454494631 >> v2:172.21.15.203:3300/0 conn(0x7fe694142dc0 0x7fe69414cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.334+0000 7fe69c33d640 1 --2- 172.21.15.106:0/454494631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6940a9e30 0x7fe694142880 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.334+0000 7fe69c33d640 1 -- 172.21.15.106:0/454494631 >> 172.21.15.106:0/454494631 conn(0x7fe6940b8030 msgr2=0x7fe6940b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:29.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.334+0000 7fe69c33d640 1 -- 172.21.15.106:0/454494631 shutdown_connections 2024-04-04T21:02:29.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.334+0000 7fe69c33d640 1 -- 172.21.15.106:0/454494631 wait complete. 2024-04-04T21:02:29.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.334+0000 7fe69c33d640 10 monclient: build_initial_monmap 2024-04-04T21:02:29.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.334+0000 7fe69c33d640 10 monclient: monmap: 2024-04-04T21:02:29.339 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:29.339 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:29.339 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:29.339 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:29.339 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:29.339 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.339 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:29.340 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:29.340 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:29.340 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 1 librados: starting msgr at 2024-04-04T21:02:29.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 1 librados: starting objecter 2024-04-04T21:02:29.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 1 Processor -- start 2024-04-04T21:02:29.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 1 -- start start 2024-04-04T21:02:29.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 1 librados: setting wanted keys 2024-04-04T21:02:29.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 1 librados: calling monclient init 2024-04-04T21:02:29.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 10 monclient: init 2024-04-04T21:02:29.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-04T21:02:29.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6940a9a50 0x7fe694138aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 10 monclient(hunting): picked mon.noname-b con 0x7fe6940a9a50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:29.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6940a9e30 0x7fe6941d79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 10 monclient(hunting): picked mon.noname-c con 0x7fe6940a9e30 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe694142dc0 0x7fe6941d7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 10 monclient(hunting): picked mon.noname-a con 0x7fe694142dc0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6941433d0 con 0x7fe694142dc0 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe694143250 con 0x7fe6940a9a50 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69414d430 con 0x7fe6940a9e30 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.335+0000 7fe69c33d640 10 monclient(hunting): authenticate will time out at 1398.602995s 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe6998d8640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6940a9e30 0x7fe6941d79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe69a8da640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe694142dc0 0x7fe6941d7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe69a0d9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6940a9a50 0x7fe694138aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe6998d8640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6940a9e30 0x7fe6941d79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:32946/0 (socket says 172.21.15.106:32946) 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe69a8da640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe694142dc0 0x7fe6941d7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34994/0 (socket says 172.21.15.106:34994) 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe6998d8640 1 -- 172.21.15.106:0/723836038 learned_addr learned my addr 172.21.15.106:0/723836038 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe6998d8640 10 monclient(hunting): get_auth_request con 0x7fe6940a9e30 auth_method 0 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe6998d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe6998d8640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe6998d8640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe69a8da640 10 monclient(hunting): get_auth_request con 0x7fe694142dc0 auth_method 0 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe69a8da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe69a8da640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe69a8da640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe69a0d9640 10 monclient(hunting): get_auth_request con 0x7fe6940a9a50 auth_method 0 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe69a0d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe69a0d9640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.336+0000 7fe69a0d9640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.337+0000 7fe6998d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.337+0000 7fe6998d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.337+0000 7fe6998d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:29.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.337+0000 7fe69a8da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.337+0000 7fe69a8da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.337+0000 7fe69a8da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:29.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.337+0000 7fe69a0d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.337+0000 7fe69a0d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.337+0000 7fe69a0d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:29.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.339+0000 7fe6998d8640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-04-04T21:02:29.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.340+0000 7fe6998d8640 1 -- 172.21.15.106:0/723836038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6940a9a50 msgr2=0x7fe694138aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.340+0000 7fe6998d8640 1 --2- 172.21.15.106:0/723836038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6940a9a50 0x7fe694138aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.340+0000 7fe6998d8640 1 -- 172.21.15.106:0/723836038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe694142dc0 msgr2=0x7fe6941d7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.340+0000 7fe6998d8640 1 --2- 172.21.15.106:0/723836038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe694142dc0 0x7fe6941d7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.340+0000 7fe6998d8640 10 monclient: _finish_hunting 0 2024-04-04T21:02:29.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.340+0000 7fe6998d8640 1 monclient: found mon.noname-c 2024-04-04T21:02:29.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.340+0000 7fe6998d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:29.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.340+0000 7fe6998d8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:02:29.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.340+0000 7fe6998d8640 1 -- 172.21.15.106:0/723836038 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69413c8a0 con 0x7fe6940a9e30 2024-04-04T21:02:29.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.342+0000 7fe6998d8640 1 --2- 172.21.15.106:0/723836038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6940a9e30 0x7fe6941d79c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe6840098c0 tx=0x7fe6840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.342+0000 7fe68affd640 10 client.?.objecter ms_handle_connect 0x7fe6940a9e30 2024-04-04T21:02:29.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.342+0000 7fe68affd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:29.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe68affd640 1 -- 172.21.15.106:0/723836038 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe684032630 con 0x7fe6940a9e30 2024-04-04T21:02:29.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe68affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe68affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:29.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe68affd640 10 monclient: dump: 2024-04-04T21:02:29.348 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.348 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe68affd640 10 monclient: _finish_auth 0 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe68affd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe68affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe69c33d640 5 monclient: authenticate success, global_id 4313 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe68affd640 1 -- 172.21.15.106:0/723836038 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6840327d0 con 0x7fe6940a9e30 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe68affd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe68affd640 1 -- 172.21.15.106:0/723836038 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe684032bc0 con 0x7fe6940a9e30 2024-04-04T21:02:29.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe68affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe68affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:29.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe68affd640 10 monclient: dump: 2024-04-04T21:02:29.350 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.350 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.350 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.350 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.350 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.350 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.350 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.350 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.350 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.350 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe6988d6640 4 set_mon_vals no callback set 2024-04-04T21:02:29.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe69c33d640 10 monclient: _renew_subs 2024-04-04T21:02:29.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe69c33d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:29.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe69c33d640 1 -- 172.21.15.106:0/723836038 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe69413cb10 con 0x7fe6940a9e30 2024-04-04T21:02:29.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe69c33d640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:29.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe69c33d640 10 monclient: _renew_subs 2024-04-04T21:02:29.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe69c33d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:29.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe69c33d640 1 -- 172.21.15.106:0/723836038 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6941422b0 con 0x7fe6940a9e30 2024-04-04T21:02:29.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.346+0000 7fe69c33d640 1 librados: init done 2024-04-04T21:02:29.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.347+0000 7fe69c33d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.347+0000 7fe69c33d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.347+0000 7fe69c33d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:29.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.347+0000 7fe69c33d640 1 -- 172.21.15.106:0/723836038 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69404f050 con 0x7fe6940a9e30 2024-04-04T21:02:29.366 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.365+0000 7fe68affd640 1 -- 172.21.15.106:0/723836038 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe68404e430 con 0x7fe6940a9e30 2024-04-04T21:02:29.366 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.365+0000 7fe68affd640 1 --2- 172.21.15.106:0/723836038 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe67806ce60 0x7fe67806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.366 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.365+0000 7fe69a0d9640 1 --2- 172.21.15.106:0/723836038 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe67806ce60 0x7fe67806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.366 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.365+0000 7fe69a0d9640 10 monclient: get_auth_request con 0x7fe67806ce60 auth_method 0 2024-04-04T21:02:29.366 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.365+0000 7fe68affd640 10 client.4313.objecter ms_handle_connect 0x7fe67806ce60 2024-04-04T21:02:29.367 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.365+0000 7fe69a0d9640 1 --2- 172.21.15.106:0/723836038 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe67806ce60 0x7fe67806f320 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe690000f60 tx=0x7fe690009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.368+0000 7fe68affd640 1 -- 172.21.15.106:0/723836038 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe68402f030 con 0x7fe6940a9e30 2024-04-04T21:02:29.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.368+0000 7fe68affd640 10 client.4313.objecter ms_dispatch 0x7fe694000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:29.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.368+0000 7fe68affd640 3 client.4313.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:29.368 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.368+0000 7fe68affd640 3 client.4313.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:29.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.368+0000 7fe68affd640 20 client.4313.objecter dump_active .. 0 homeless 2024-04-04T21:02:29.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.375+0000 7f19ebfe5640 10 monclient: get_monmap_and_config 2024-04-04T21:02:29.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.375+0000 7f19ebfe5640 10 monclient: build_initial_monmap 2024-04-04T21:02:29.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.375+0000 7f19ebfe5640 10 monclient: monmap: 2024-04-04T21:02:29.376 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:29.376 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.376 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:29.376704+0000 2024-04-04T21:02:29.376 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:29.376704+0000 2024-04-04T21:02:29.376 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:29.376 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.376 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:29.376 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:29.376 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:29.377 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:29.377 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:29.377 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 1 Processor -- start 2024-04-04T21:02:29.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 1 -- start start 2024-04-04T21:02:29.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 10 monclient: init 2024-04-04T21:02:29.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f19e40a9a50 0x7f19e40a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 10 monclient(hunting): picked mon.noname-c con 0x7f19e40a9a50 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19e413f6a0 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19e413fac0 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e413ff30 con 0x7f19e413fac0 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e40ab750 con 0x7f19e413f6a0 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e40aa370 con 0x7f19e40a9a50 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.376+0000 7f19ebfe5640 10 monclient(hunting): authenticate will time out at 1398.644159s 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.377+0000 7f19ea582640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f19e413fac0 0x7f19e414cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:49556/0 (socket says 172.21.15.106:49556) 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.377+0000 7f19ea582640 1 -- 172.21.15.106:0/42182733 learned_addr learned my addr 172.21.15.106:0/42182733 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.377+0000 7f19e9d81640 1 --2- 172.21.15.106:0/42182733 >> v2:172.21.15.203:3300/0 conn(0x7f19e40a9a50 0x7f19e40a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.378+0000 7f19e9d81640 10 monclient(hunting): get_auth_request con 0x7f19e40a9a50 auth_method 0 2024-04-04T21:02:29.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.378+0000 7f19e9d81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.378+0000 7f19e9d81640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.378+0000 7f19e9d81640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.379+0000 7f19e9d81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.379+0000 7f19e9d81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.379+0000 7f19e9d81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:29.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.379+0000 7f19e8d7f640 1 -- 172.21.15.106:0/42182733 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489527923 0 0) 0x7f19e413ff30 con 0x7f19e413fac0 2024-04-04T21:02:29.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.379+0000 7f19e8d7f640 10 monclient(hunting): _negotiate 2024-04-04T21:02:29.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.379+0000 7f19e8d7f640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.379+0000 7f19e8d7f640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.379+0000 7f19e8d7f640 10 monclient(hunting): my global_id is 4341 2024-04-04T21:02:29.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.379+0000 7f19e8d7f640 1 -- 172.21.15.106:0/42182733 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d0003520 con 0x7f19e413fac0 2024-04-04T21:02:29.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.381+0000 7f19e9d81640 10 monclient(hunting): handle_auth_done global_id 4381 payload 274 2024-04-04T21:02:29.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.381+0000 7f19e9d81640 1 -- 172.21.15.106:0/42182733 >> v1:172.21.15.203:6789/0 conn(0x7f19e413f6a0 legacy=0x7f19e4142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.381+0000 7f19e9d81640 1 -- 172.21.15.106:0/42182733 >> v1:172.21.15.106:6789/0 conn(0x7f19e413fac0 legacy=0x7f19e414cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.381+0000 7f19e9d81640 10 monclient: _finish_hunting 0 2024-04-04T21:02:29.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.381+0000 7f19e9d81640 1 monclient: found mon.noname-c 2024-04-04T21:02:29.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.381+0000 7f19e9d81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:29.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.381+0000 7f19e9d81640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:02:29.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.381+0000 7f19e9d81640 1 -- 172.21.15.106:0/42182733 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e4140120 con 0x7f19e40a9a50 2024-04-04T21:02:29.385 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.385+0000 7f19e9d81640 1 --2- 172.21.15.106:0/42182733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e40a9a50 0x7f19e40a9e30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f19e00097a0 tx=0x7f19e002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df29cc8865c26e5b server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19e8d7f640 1 -- 172.21.15.106:0/42182733 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0035020 con 0x7f19e40a9a50 2024-04-04T21:02:29.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19e8d7f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19e8d7f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:29.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19e8d7f640 10 monclient: dump: 2024-04-04T21:02:29.390 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.390 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.390 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.390 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.390 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.390 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.390 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.390 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.391 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.391 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19e8d7f640 10 monclient: _finish_auth 0 2024-04-04T21:02:29.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19e8d7f640 10 monclient: _check_auth_tickets 2024-04-04T21:02:29.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19e8d7f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:29.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19e8d7f640 1 -- 172.21.15.106:0/42182733 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e0031830 con 0x7f19e40a9a50 2024-04-04T21:02:29.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19e8d7f640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:29.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19e8d7f640 1 -- 172.21.15.106:0/42182733 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0031c00 con 0x7f19e40a9a50 2024-04-04T21:02:29.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19e8d7f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19e8d7f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:29.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19e8d7f640 10 monclient: dump: 2024-04-04T21:02:29.392 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.392 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.392 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.392 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.392 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.392 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.392 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.392 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.392 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.392 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.393 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19ebfe5640 5 monclient: authenticate success, global_id 4381 2024-04-04T21:02:29.393 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19ebfe5640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:29.393 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.389+0000 7f19ebfe5640 4 set_mon_vals no callback set 2024-04-04T21:02:29.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.390+0000 7f19ebfe5640 10 monclient: shutdown 2024-04-04T21:02:29.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.390+0000 7f19ebfe5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:29.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.390+0000 7f19ebfe5640 1 -- 172.21.15.106:0/42182733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e40a9a50 msgr2=0x7f19e40a9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.390+0000 7f19ebfe5640 1 --2- 172.21.15.106:0/42182733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e40a9a50 0x7f19e40a9e30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f19e00097a0 tx=0x7f19e002b8d0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.390+0000 7f19ebfe5640 1 -- 172.21.15.106:0/42182733 shutdown_connections 2024-04-04T21:02:29.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.390+0000 7f19ebfe5640 1 --2- 172.21.15.106:0/42182733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e40a9a50 0x7f19e40a9e30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.390+0000 7f19ebfe5640 1 -- 172.21.15.106:0/42182733 >> 172.21.15.106:0/42182733 conn(0x7f19e40b8030 msgr2=0x7f19e40b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:29.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.390+0000 7f19ebfe5640 1 -- 172.21.15.106:0/42182733 shutdown_connections 2024-04-04T21:02:29.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.390+0000 7f19ebfe5640 1 -- 172.21.15.106:0/42182733 wait complete. 2024-04-04T21:02:29.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.390+0000 7f19ebfe5640 10 monclient: build_initial_monmap 2024-04-04T21:02:29.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.390+0000 7f19ebfe5640 10 monclient: monmap: 2024-04-04T21:02:29.398 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:29.398 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:29.399 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:29.399 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:29.399 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:29.399 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.399 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:29.399 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:29.399 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:29.400 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.400 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 1 librados: starting msgr at 2024-04-04T21:02:29.400 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 1 librados: starting objecter 2024-04-04T21:02:29.400 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 1 Processor -- start 2024-04-04T21:02:29.400 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 1 -- start start 2024-04-04T21:02:29.400 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 1 librados: setting wanted keys 2024-04-04T21:02:29.400 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 1 librados: calling monclient init 2024-04-04T21:02:29.400 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 10 monclient: init 2024-04-04T21:02:29.400 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.400 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:02:29.400 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19e40a9a50 0x7f19e41cf200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 10 monclient(hunting): picked mon.noname-a con 0x7f19e40a9a50 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:29.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19e413f6a0 0x7f19e41cca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 10 monclient(hunting): picked mon.noname-c con 0x7f19e413f6a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:29.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e413fac0 0x7f19e41ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 10 monclient(hunting): picked mon.noname-b con 0x7f19e413fac0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:29.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e4143110 con 0x7f19e40a9a50 2024-04-04T21:02:29.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e4142f90 con 0x7f19e413fac0 2024-04-04T21:02:29.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e4143290 con 0x7f19e413f6a0 2024-04-04T21:02:29.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.391+0000 7f19ebfe5640 10 monclient(hunting): authenticate will time out at 1398.658951s 2024-04-04T21:02:29.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19e9d81640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19e40a9a50 0x7f19e41cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19ea582640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e413fac0 0x7f19e41ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19e9d81640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19e40a9a50 0x7f19e41cf200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34996/0 (socket says 172.21.15.106:34996) 2024-04-04T21:02:29.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19e9d81640 1 -- 172.21.15.106:0/1581971477 learned_addr learned my addr 172.21.15.106:0/1581971477 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:29.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19e9d81640 10 monclient(hunting): get_auth_request con 0x7f19e40a9a50 auth_method 0 2024-04-04T21:02:29.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19e9d81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19e9d81640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19e9d81640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19ea582640 10 monclient(hunting): get_auth_request con 0x7f19e413fac0 auth_method 0 2024-04-04T21:02:29.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19ea582640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19ea582640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19ea582640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19e9d81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19e9d81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.392+0000 7f19e9d81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:29.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.393+0000 7f19ea582640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.393+0000 7f19ea582640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.393+0000 7f19ea582640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:29.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.395+0000 7f19e9d81640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2024-04-04T21:02:29.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.395+0000 7f19e9d81640 1 -- 172.21.15.106:0/1581971477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19e413f6a0 msgr2=0x7f19e41cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.395+0000 7f19e9d81640 1 --2- 172.21.15.106:0/1581971477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19e413f6a0 0x7f19e41cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.395+0000 7f19e9d81640 1 -- 172.21.15.106:0/1581971477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e413fac0 msgr2=0x7f19e41ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.395+0000 7f19e9d81640 1 --2- 172.21.15.106:0/1581971477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e413fac0 0x7f19e41ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.395+0000 7f19e9d81640 10 monclient: _finish_hunting 0 2024-04-04T21:02:29.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.395+0000 7f19e9d81640 1 monclient: found mon.noname-a 2024-04-04T21:02:29.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.395+0000 7f19e9d81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:29.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.395+0000 7f19e9d81640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:29.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.395+0000 7f19e9d81640 1 -- 172.21.15.106:0/1581971477 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e41cf740 con 0x7f19e40a9a50 2024-04-04T21:02:29.405 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-04T21:02:29.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.398+0000 7f19e9d81640 1 --2- 172.21.15.106:0/1581971477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19e40a9a50 0x7f19e41cf200 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f19e002bee0 tx=0x7f19e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.398+0000 7f19da7fc640 10 client.?.objecter ms_handle_connect 0x7f19e40a9a50 2024-04-04T21:02:29.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.398+0000 7f19da7fc640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:29.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19da7fc640 1 -- 172.21.15.106:0/1581971477 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0035040 con 0x7f19e40a9a50 2024-04-04T21:02:29.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:29.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19da7fc640 10 monclient: dump: 2024-04-04T21:02:29.408 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.409 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.409 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.409 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.409 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.409 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.409 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.409 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.409 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.409 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19da7fc640 10 monclient: _finish_auth 0 2024-04-04T21:02:29.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19da7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:02:29.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:29.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19da7fc640 1 -- 172.21.15.106:0/1581971477 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e003b920 con 0x7f19e40a9a50 2024-04-04T21:02:29.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19ebfe5640 5 monclient: authenticate success, global_id 4344 2024-04-04T21:02:29.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19ebfe5640 10 monclient: _renew_subs 2024-04-04T21:02:29.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19ebfe5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:29.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19ebfe5640 1 -- 172.21.15.106:0/1581971477 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e41ce790 con 0x7f19e40a9a50 2024-04-04T21:02:29.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19da7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:29.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19da7fc640 1 -- 172.21.15.106:0/1581971477 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e003bd10 con 0x7f19e40a9a50 2024-04-04T21:02:29.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:29.410 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19da7fc640 10 monclient: dump: 2024-04-04T21:02:29.410 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.411 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.411 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.411 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.411 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.411 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.411 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.411 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.411 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.411 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19ebfe5640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:29.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19ebfe5640 10 monclient: _renew_subs 2024-04-04T21:02:29.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19db7fe640 4 set_mon_vals no callback set 2024-04-04T21:02:29.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19ebfe5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:29.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19ebfe5640 1 -- 172.21.15.106:0/1581971477 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e41cc540 con 0x7f19e40a9a50 2024-04-04T21:02:29.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.401+0000 7f19ebfe5640 1 librados: init done 2024-04-04T21:02:29.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.402+0000 7f19ebfe5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.402+0000 7f19ebfe5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.402+0000 7f19ebfe5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:29.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.402+0000 7f19ebfe5640 1 -- 172.21.15.106:0/1581971477 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e404f050 con 0x7f19e40a9a50 2024-04-04T21:02:29.412 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-04T21:02:29.441 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-04T21:02:29.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.517+0000 7fe3f1667640 10 monclient: get_monmap_and_config 2024-04-04T21:02:29.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.517+0000 7fe3f1667640 10 monclient: build_initial_monmap 2024-04-04T21:02:29.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.517+0000 7fe3f1667640 10 monclient: monmap: 2024-04-04T21:02:29.519 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:29.519 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.519 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:29.518233+0000 2024-04-04T21:02:29.519 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:29.518233+0000 2024-04-04T21:02:29.519 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:29.519 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.520 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:29.520 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:29.520 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:29.520 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:29.520 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:29.520 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.517+0000 7fe3f1667640 1 Processor -- start 2024-04-04T21:02:29.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.517+0000 7fe3f1667640 1 -- start start 2024-04-04T21:02:29.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.517+0000 7fe3f1667640 10 monclient: init 2024-04-04T21:02:29.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.518+0000 7fe3f1667640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.518+0000 7fe3f1667640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-04-04T21:02:29.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.518+0000 7fe3f1667640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe3ec0a9a50 0x7fe3ec0a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.518+0000 7fe3f1667640 10 monclient(hunting): picked mon.noname-a con 0x7fe3ec0a9a50 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:29.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.518+0000 7fe3f1667640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3ec13f6a0 0x7fe3ec13fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.518+0000 7fe3f1667640 10 monclient(hunting): picked mon.noname-b con 0x7fe3ec13f6a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:29.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.518+0000 7fe3f1667640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3ec142880 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:29.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.518+0000 7fe3f1667640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ec143420 con 0x7fe3ec142880 2024-04-04T21:02:29.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.518+0000 7fe3f1667640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.518+0000 7fe3f1667640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ec0aa370 con 0x7fe3ec0a9a50 2024-04-04T21:02:29.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.518+0000 7fe3f1667640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.518+0000 7fe3f1667640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec0ab750 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.518+0000 7fe3f1667640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.518+0000 7fe3f1667640 10 monclient(hunting): authenticate will time out at 1398.785909s 2024-04-04T21:02:29.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.527+0000 7f5fecca5640 10 monclient: get_monmap_and_config 2024-04-04T21:02:29.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.527+0000 7f5fecca5640 10 monclient: build_initial_monmap 2024-04-04T21:02:29.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.527+0000 7f5fecca5640 10 monclient: monmap: 2024-04-04T21:02:29.528 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:29.528 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.528 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:29.528364+0000 2024-04-04T21:02:29.528 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:29.528364+0000 2024-04-04T21:02:29.528 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:29.528 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.528 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:29.528 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:29.528 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:29.528 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:29.528 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:29.528 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.527+0000 7f5fecca5640 1 Processor -- start 2024-04-04T21:02:29.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.528+0000 7f5fecca5640 1 -- start start 2024-04-04T21:02:29.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.528+0000 7f5fecca5640 10 monclient: init 2024-04-04T21:02:29.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.528+0000 7f5fecca5640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.528+0000 7f5fecca5640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-04-04T21:02:29.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.528+0000 7f5fecca5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5fe80a9a50 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:29.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.528+0000 7f5fecca5640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fe80a9e30 0x7f5fe8142880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.528+0000 7f5fecca5640 10 monclient(hunting): picked mon.noname-b con 0x7f5fe80a9e30 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:29.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.528+0000 7f5fecca5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5fe8142dc0 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:29.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.529+0000 7f5fecca5640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe80ab750 con 0x7f5fe8142dc0 2024-04-04T21:02:29.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.529+0000 7f5fecca5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe8140120 con 0x7f5fe80a9a50 2024-04-04T21:02:29.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.529+0000 7f5fecca5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.529+0000 7f5fecca5640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe80aa660 con 0x7f5fe80a9e30 2024-04-04T21:02:29.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.529+0000 7f5fecca5640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.529+0000 7f5fecca5640 10 monclient(hunting): authenticate will time out at 1398.796314s 2024-04-04T21:02:29.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.529+0000 7f5fe659c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f5fe80a9a50 0x7f5fe813f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:41938/0 (socket says 172.21.15.106:41938) 2024-04-04T21:02:29.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.529+0000 7f5fe659c640 1 -- 172.21.15.106:0/4071819487 learned_addr learned my addr 172.21.15.106:0/4071819487 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:29.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.532+0000 7f5fe559a640 1 -- 172.21.15.106:0/4071819487 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696768521 0 0) 0x7f5fe8140120 con 0x7f5fe80a9a50 2024-04-04T21:02:29.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.532+0000 7f5fe559a640 10 monclient(hunting): _negotiate 2024-04-04T21:02:29.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.532+0000 7f5fe559a640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.532+0000 7f5fe559a640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.532+0000 7f5fe559a640 10 monclient(hunting): my global_id is 4387 2024-04-04T21:02:29.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.532+0000 7f5fe559a640 1 -- 172.21.15.106:0/4071819487 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd0003520 con 0x7f5fe80a9a50 2024-04-04T21:02:29.536 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 -- 172.21.15.106:0/4071819487 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1948509535 0 0) 0x7f5fd0003520 con 0x7f5fe80a9a50 2024-04-04T21:02:29.536 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 -- 172.21.15.106:0/4071819487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fe80a9e30 msgr2=0x7f5fe8142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:29.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 --2- 172.21.15.106:0/4071819487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fe80a9e30 0x7f5fe8142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 -- 172.21.15.106:0/4071819487 >> v1:172.21.15.106:6789/0 conn(0x7f5fe8142dc0 legacy=0x7f5fe814cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:29.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient: _finish_hunting 0 2024-04-04T21:02:29.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 monclient: found mon.noname-c-legacy 2024-04-04T21:02:29.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:29.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-04-04T21:02:29.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 -- 172.21.15.106:0/4071819487 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe80aa250 con 0x7f5fe80a9a50 2024-04-04T21:02:29.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient: _finish_auth 0 2024-04-04T21:02:29.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient: _check_auth_tickets 2024-04-04T21:02:29.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 -- 172.21.15.106:0/4071819487 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 141466616 0 0) 0x7f5fd8001670 con 0x7f5fe80a9a50 2024-04-04T21:02:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-04T21:02:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient: dump: 2024-04-04T21:02:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.538 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-04-04T21:02:29.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 -- 172.21.15.106:0/4071819487 >> v1:172.21.15.203:6789/0 conn(0x7f5fe80a9a50 legacy=0x7f5fe813f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-04T21:02:29.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 --2- 172.21.15.106:0/4071819487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fe80a9e30 0x7f5fd00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient(hunting): picked mon.c con 0x7f5fe80a9e30 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:29.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 --2- 172.21.15.106:0/4071819487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5fd00040f0 0x7f5fd0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient(hunting): picked mon.a con 0x7f5fd00040f0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:29.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 --2- 172.21.15.106:0/4071819487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fd0004e60 0x7f5fd000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient(hunting): picked mon.b con 0x7f5fd0004e60 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:29.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 -- 172.21.15.106:0/4071819487 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe80aa660 con 0x7f5fd00040f0 2024-04-04T21:02:29.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 -- 172.21.15.106:0/4071819487 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd00055e0 con 0x7f5fd0004e60 2024-04-04T21:02:29.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 1 -- 172.21.15.106:0/4071819487 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd000bf00 con 0x7f5fe80a9e30 2024-04-04T21:02:29.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.535+0000 7f5fe559a640 10 monclient(hunting): _finish_auth 0 2024-04-04T21:02:29.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.538+0000 7f5fe6d9d640 1 --2- 172.21.15.106:0/4071819487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fd0004e60 0x7f5fd000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.539+0000 7f5fe6d9d640 10 monclient(hunting): get_auth_request con 0x7f5fd0004e60 auth_method 0 2024-04-04T21:02:29.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.539+0000 7f5fe6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.539+0000 7f5fe6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.539+0000 7f5fe6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-04T21:02:29.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.540+0000 7f5fe6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.540+0000 7f5fe6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.540+0000 7f5fe6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-04T21:02:29.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fe6d9d640 10 monclient(hunting): handle_auth_done global_id 4387 payload 293 2024-04-04T21:02:29.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fe6d9d640 1 -- 172.21.15.106:0/4071819487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fe80a9e30 msgr2=0x7f5fd00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:29.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fe6d9d640 1 --2- 172.21.15.106:0/4071819487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fe80a9e30 0x7f5fd00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fe6d9d640 1 -- 172.21.15.106:0/4071819487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5fd00040f0 msgr2=0x7f5fd0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:29.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fe6d9d640 1 --2- 172.21.15.106:0/4071819487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5fd00040f0 0x7f5fd0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fe6d9d640 10 monclient: _finish_hunting 0 2024-04-04T21:02:29.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fe6d9d640 1 monclient: found mon.b 2024-04-04T21:02:29.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fe6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:29.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fe6d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:29.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fe6d9d640 1 -- 172.21.15.106:0/4071819487 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd0003520 con 0x7f5fd0004e60 2024-04-04T21:02:29.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fe6d9d640 10 monclient: _finish_auth 0 2024-04-04T21:02:29.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fe6d9d640 10 monclient: _check_auth_tickets 2024-04-04T21:02:29.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fe6d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:29.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fecca5640 5 monclient: authenticate success, global_id 4387 2024-04-04T21:02:29.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.541+0000 7f5fecca5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:29.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.543+0000 7f5fe6d9d640 1 --2- 172.21.15.106:0/4071819487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fd0004e60 0x7f5fd000b9c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0001d50 tx=0x7f5fd0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f40aaddc021704b server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.549 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.547+0000 7f5fe559a640 1 -- 172.21.15.106:0/4071819487 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0001920 con 0x7f5fd0004e60 2024-04-04T21:02:29.549 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.547+0000 7f5fe559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.549 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.547+0000 7f5fe559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:29.549 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.547+0000 7f5fe559a640 10 monclient: dump: 2024-04-04T21:02:29.549 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.549 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.549 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.549 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.549 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.549 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.550 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.550 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.550 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.550 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.547+0000 7f5fe559a640 1 -- 172.21.15.106:0/4071819487 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe0035070 con 0x7f5fd0004e60 2024-04-04T21:02:29.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.547+0000 7f5fecca5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:29.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.547+0000 7f5fe559a640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:29.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.547+0000 7f5fe559a640 1 -- 172.21.15.106:0/4071819487 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0003b90 con 0x7f5fd0004e60 2024-04-04T21:02:29.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.547+0000 7f5fe559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.547+0000 7f5fe559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:29.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.547+0000 7f5fe559a640 10 monclient: dump: 2024-04-04T21:02:29.550 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.550 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.550 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.551 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.551 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.551 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.551 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.551 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.551 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.551 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.547+0000 7f5fecca5640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:29.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.547+0000 7f5fecca5640 4 set_mon_vals no callback set 2024-04-04T21:02:29.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.548+0000 7f5fecca5640 10 monclient: shutdown 2024-04-04T21:02:29.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.548+0000 7f5fecca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:29.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.548+0000 7f5fecca5640 1 -- 172.21.15.106:0/4071819487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fd0004e60 msgr2=0x7f5fd000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.548+0000 7f5fecca5640 1 --2- 172.21.15.106:0/4071819487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fd0004e60 0x7f5fd000b9c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0001d50 tx=0x7f5fd0005830 comp rx=0 tx=0).stop 2024-04-04T21:02:29.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.548+0000 7f5fe6d9d640 1 -- 172.21.15.106:0/4071819487 reap_dead start 2024-04-04T21:02:29.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.548+0000 7f5fecca5640 1 -- 172.21.15.106:0/4071819487 shutdown_connections 2024-04-04T21:02:29.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.548+0000 7f5fecca5640 1 -- 172.21.15.106:0/4071819487 >> 172.21.15.106:0/4071819487 conn(0x7f5fe80b8030 msgr2=0x7f5fe80b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:29.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.548+0000 7f5fecca5640 1 -- 172.21.15.106:0/4071819487 shutdown_connections 2024-04-04T21:02:29.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.548+0000 7f5fecca5640 1 -- 172.21.15.106:0/4071819487 wait complete. 2024-04-04T21:02:29.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.549+0000 7f5fecca5640 10 monclient: build_initial_monmap 2024-04-04T21:02:29.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.549+0000 7f5fecca5640 10 monclient: monmap: 2024-04-04T21:02:29.555 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:29.555 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:29.555 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:29.555 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:29.555 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:29.555 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.555 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:29.555 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:29.555 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:29.555 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.550+0000 7f5fecca5640 1 librados: starting msgr at 2024-04-04T21:02:29.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.550+0000 7f5fecca5640 1 librados: starting objecter 2024-04-04T21:02:29.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.550+0000 7f5fecca5640 1 Processor -- start 2024-04-04T21:02:29.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.550+0000 7f5fecca5640 1 -- start start 2024-04-04T21:02:29.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.550+0000 7f5fecca5640 1 librados: setting wanted keys 2024-04-04T21:02:29.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.550+0000 7f5fecca5640 1 librados: calling monclient init 2024-04-04T21:02:29.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.550+0000 7f5fecca5640 10 monclient: init 2024-04-04T21:02:29.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-04T21:02:29.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fd0004e60 0x7f5fe80ac080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 10 monclient(hunting): picked mon.noname-c con 0x7f5fd0004e60 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:29.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5fe8146620 0x7f5fe81d7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 10 monclient(hunting): picked mon.noname-a con 0x7f5fe8146620 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe81d7fa0 0x7f5fe81daac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 10 monclient(hunting): picked mon.noname-b con 0x7f5fe81d7fa0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe81433d0 con 0x7f5fe8146620 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe8143250 con 0x7f5fe81d7fa0 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe8142880 con 0x7f5fd0004e60 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.551+0000 7f5fecca5640 10 monclient(hunting): authenticate will time out at 1398.818788s 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.552+0000 7f5fe6d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe81d7fa0 0x7f5fe81daac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.552+0000 7f5fe6d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe81d7fa0 0x7f5fe81daac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50380/0 (socket says 172.21.15.106:50380) 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.552+0000 7f5fe6d9d640 1 -- 172.21.15.106:0/3900206976 learned_addr learned my addr 172.21.15.106:0/3900206976 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.552+0000 7f5fe6d9d640 10 monclient(hunting): get_auth_request con 0x7f5fe81d7fa0 auth_method 0 2024-04-04T21:02:29.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.552+0000 7f5fe6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.552+0000 7f5fe6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.552+0000 7f5fe6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.553+0000 7f5fe6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.553+0000 7f5fe6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.553+0000 7f5fe6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:29.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.555+0000 7f5fe6d9d640 10 monclient(hunting): handle_auth_done global_id 4390 payload 824 2024-04-04T21:02:29.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.555+0000 7f5fe6d9d640 1 -- 172.21.15.106:0/3900206976 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fd0004e60 msgr2=0x7f5fe80ac080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.555+0000 7f5fe6d9d640 1 --2- 172.21.15.106:0/3900206976 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fd0004e60 0x7f5fe80ac080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.555+0000 7f5fe6d9d640 1 -- 172.21.15.106:0/3900206976 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5fe8146620 msgr2=0x7f5fe81d7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.555+0000 7f5fe6d9d640 1 --2- 172.21.15.106:0/3900206976 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5fe8146620 0x7f5fe81d7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.555+0000 7f5fe6d9d640 10 monclient: _finish_hunting 0 2024-04-04T21:02:29.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.555+0000 7f5fe6d9d640 1 monclient: found mon.noname-b 2024-04-04T21:02:29.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.555+0000 7f5fe6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:29.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.555+0000 7f5fe6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:29.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.555+0000 7f5fe6d9d640 1 -- 172.21.15.106:0/3900206976 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe81d8430 con 0x7f5fe81d7fa0 2024-04-04T21:02:29.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.558+0000 7f5fe6d9d640 1 --2- 172.21.15.106:0/3900206976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe81d7fa0 0x7f5fe81daac0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0032040 tx=0x7f5fe0001330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.558+0000 7f5fd6ffd640 10 client.?.objecter ms_handle_connect 0x7f5fe81d7fa0 2024-04-04T21:02:29.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.558+0000 7f5fd6ffd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:29.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fd6ffd640 1 -- 172.21.15.106:0/3900206976 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0004e10 con 0x7f5fe81d7fa0 2024-04-04T21:02:29.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fd6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:29.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fd6ffd640 10 monclient: dump: 2024-04-04T21:02:29.562 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.562 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.562 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.562 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.562 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.562 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.563 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.563 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.563 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.563 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fd6ffd640 10 monclient: _finish_auth 0 2024-04-04T21:02:29.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fd6ffd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:29.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:29.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fd6ffd640 1 -- 172.21.15.106:0/3900206976 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe0005070 con 0x7f5fe81d7fa0 2024-04-04T21:02:29.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fd6ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:29.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fd6ffd640 1 -- 172.21.15.106:0/3900206976 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0005440 con 0x7f5fe81d7fa0 2024-04-04T21:02:29.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fd6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:29.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fd6ffd640 10 monclient: dump: 2024-04-04T21:02:29.566 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.566 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.566 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.566 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.566 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.566 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.566 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.566 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.566 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.566 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fe4d99640 4 set_mon_vals no callback set 2024-04-04T21:02:29.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fecca5640 5 monclient: authenticate success, global_id 4390 2024-04-04T21:02:29.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fecca5640 10 monclient: _renew_subs 2024-04-04T21:02:29.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fecca5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:29.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fecca5640 1 -- 172.21.15.106:0/3900206976 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe80ad9c0 con 0x7f5fe81d7fa0 2024-04-04T21:02:29.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fecca5640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:29.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fecca5640 10 monclient: _renew_subs 2024-04-04T21:02:29.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fecca5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:29.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fecca5640 1 -- 172.21.15.106:0/3900206976 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe80ac6f0 con 0x7f5fe81d7fa0 2024-04-04T21:02:29.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.561+0000 7f5fecca5640 1 librados: init done 2024-04-04T21:02:29.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.563+0000 7f5fecca5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.563+0000 7f5fecca5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.563+0000 7f5fecca5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:29.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.563+0000 7f5fecca5640 1 -- 172.21.15.106:0/3900206976 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe80ad7c0 con 0x7f5fe81d7fa0 2024-04-04T21:02:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.581+0000 7f421a506640 10 monclient: get_monmap_and_config 2024-04-04T21:02:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.581+0000 7f421a506640 10 monclient: build_initial_monmap 2024-04-04T21:02:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.581+0000 7f421a506640 10 monclient: monmap: 2024-04-04T21:02:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:29.582675+0000 2024-04-04T21:02:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:29.582675+0000 2024-04-04T21:02:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:29.582 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:29.583 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:29.583 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:29.583 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:29.583 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.582+0000 7f421a506640 1 Processor -- start 2024-04-04T21:02:29.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.582+0000 7f421a506640 1 -- start start 2024-04-04T21:02:29.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.582+0000 7f421a506640 10 monclient: init 2024-04-04T21:02:29.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.583+0000 7f421a506640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.583+0000 7f421a506640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-04-04T21:02:29.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.583+0000 7f421a506640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42140a9b60 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:29.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.583+0000 7f421a506640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f42141426f0 0x7f42140b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.583+0000 7f421a506640 10 monclient(hunting): picked mon.noname-c con 0x7f42141426f0 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:29.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.583+0000 7f421a506640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f421413f810 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:29.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.583+0000 7f421a506640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42140b2e00 con 0x7f421413f810 2024-04-04T21:02:29.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.583+0000 7f421a506640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42140b8c90 con 0x7f42140a9b60 2024-04-04T21:02:29.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.583+0000 7f421a506640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.583+0000 7f421a506640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42140a9f40 con 0x7f42141426f0 2024-04-04T21:02:29.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.583+0000 7f421a506640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.583+0000 7f421a506640 10 monclient(hunting): authenticate will time out at 1398.850520s 2024-04-04T21:02:29.586 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.583+0000 7f5fd6ffd640 1 -- 172.21.15.106:0/3900206976 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5fe00055e0 con 0x7f5fe81d7fa0 2024-04-04T21:02:29.586 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.584+0000 7f5fd6ffd640 1 --2- 172.21.15.106:0/3900206976 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5fc806ceb0 0x7f5fc806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.586 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.584+0000 7f5fe659c640 1 --2- 172.21.15.106:0/3900206976 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5fc806ceb0 0x7f5fc806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.586 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.584+0000 7f5fe659c640 10 monclient: get_auth_request con 0x7f5fc806ceb0 auth_method 0 2024-04-04T21:02:29.586 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.585+0000 7f5fd6ffd640 10 client.4390.objecter ms_handle_connect 0x7f5fc806ceb0 2024-04-04T21:02:29.586 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.585+0000 7f5fe659c640 1 --2- 172.21.15.106:0/3900206976 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5fc806ceb0 0x7f5fc806f370 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8001eb0 tx=0x7f5fd8001600 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.626 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.626+0000 7fe68affd640 1 -- 172.21.15.106:0/723836038 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe684040070 con 0x7fe6940a9e30 2024-04-04T21:02:29.626 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.626+0000 7fe68affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.627 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.626+0000 7fe68affd640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:29.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.627+0000 7fe3ea7fc640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3ec13f6a0 0x7fe3ec13fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.629+0000 7fe3ea7fc640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3ec13f6a0 0x7fe3ec13fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:32960/0 (socket says 172.21.15.106:32960) 2024-04-04T21:02:29.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.629+0000 7fe3ea7fc640 1 -- 172.21.15.106:0/984373035 learned_addr learned my addr 172.21.15.106:0/984373035 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:29.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.629+0000 7fe3ea7fc640 10 monclient(hunting): get_auth_request con 0x7fe3ec13f6a0 auth_method 0 2024-04-04T21:02:29.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.629+0000 7fe3ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.629+0000 7fe3ea7fc640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.629+0000 7fe3ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.630+0000 7fe3ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.630+0000 7fe3ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.630+0000 7fe3ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:29.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.634+0000 7fe3ea7fc640 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2024-04-04T21:02:29.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.634+0000 7fe3ea7fc640 1 -- 172.21.15.106:0/984373035 >> v2:172.21.15.106:3300/0 conn(0x7fe3ec0a9a50 msgr2=0x7fe3ec0a9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:29.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.634+0000 7fe3ea7fc640 1 --2- 172.21.15.106:0/984373035 >> v2:172.21.15.106:3300/0 conn(0x7fe3ec0a9a50 0x7fe3ec0a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.634+0000 7fe3ea7fc640 1 -- 172.21.15.106:0/984373035 >> v1:172.21.15.106:6789/0 conn(0x7fe3ec142880 legacy=0x7fe3ec142d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:29.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.634+0000 7fe3ea7fc640 10 monclient: _finish_hunting 0 2024-04-04T21:02:29.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.634+0000 7fe3ea7fc640 1 monclient: found mon.noname-b 2024-04-04T21:02:29.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.635+0000 7fe3ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:29.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.635+0000 7fe3ea7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:29.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.635+0000 7fe3ea7fc640 1 -- 172.21.15.106:0/984373035 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec140120 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.638+0000 7fe3ea7fc640 1 --2- 172.21.15.106:0/984373035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3ec13f6a0 0x7fe3ec13fac0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009870 tx=0x7fe3e002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4f24a4f4a12f047 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.643 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3e9ffb640 1 -- 172.21.15.106:0/984373035 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e00317c0 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.643 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.643 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3e9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:29.643 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3e9ffb640 10 monclient: dump: 2024-04-04T21:02:29.643 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.644 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.644 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.644 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.644 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.644 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.644 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.644 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.644 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.644 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.644 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3e9ffb640 10 monclient: _finish_auth 0 2024-04-04T21:02:29.644 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3e9ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:02:29.644 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:29.644 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3e9ffb640 1 -- 172.21.15.106:0/984373035 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e0037070 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.644 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3f1667640 5 monclient: authenticate success, global_id 4316 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3f1667640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3e9ffb640 1 -- 172.21.15.106:0/984373035 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0031b80 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3f1667640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3f1667640 4 set_mon_vals no callback set 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.642+0000 7fe3e9ffb640 10 monclient: dump: 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.645 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.646 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.646 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.646 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.643+0000 7fe3f1667640 10 monclient: shutdown 2024-04-04T21:02:29.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.643+0000 7fe3f1667640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:29.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.643+0000 7fe3f1667640 1 -- 172.21.15.106:0/984373035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3ec13f6a0 msgr2=0x7fe3ec13fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.643+0000 7fe3f1667640 1 --2- 172.21.15.106:0/984373035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3ec13f6a0 0x7fe3ec13fac0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009870 tx=0x7fe3e002b860 comp rx=0 tx=0).stop 2024-04-04T21:02:29.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.643+0000 7fe3f1667640 1 -- 172.21.15.106:0/984373035 shutdown_connections 2024-04-04T21:02:29.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.643+0000 7fe3f1667640 1 --2- 172.21.15.106:0/984373035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3ec13f6a0 0x7fe3ec13fac0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.643+0000 7fe3f1667640 1 --2- 172.21.15.106:0/984373035 >> v2:172.21.15.106:3300/0 conn(0x7fe3ec0a9a50 0x7fe3ec0a9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.643+0000 7fe3f1667640 1 -- 172.21.15.106:0/984373035 >> 172.21.15.106:0/984373035 conn(0x7fe3ec0b8030 msgr2=0x7fe3ec0b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:29.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.644+0000 7fe3f1667640 1 -- 172.21.15.106:0/984373035 shutdown_connections 2024-04-04T21:02:29.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.644+0000 7fe3f1667640 1 -- 172.21.15.106:0/984373035 wait complete. 2024-04-04T21:02:29.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.644+0000 7fe3f1667640 10 monclient: build_initial_monmap 2024-04-04T21:02:29.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.644+0000 7fe3f1667640 10 monclient: monmap: 2024-04-04T21:02:29.649 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:29.649 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:29.649 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:29.649 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:29.649 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:29.649 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.649 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:29.649 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:29.649 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:29.649 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.645+0000 7fe3f1667640 1 librados: starting msgr at 2024-04-04T21:02:29.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.645+0000 7fe3f1667640 1 librados: starting objecter 2024-04-04T21:02:29.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.645+0000 7fe3f1667640 1 Processor -- start 2024-04-04T21:02:29.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.645+0000 7fe3f1667640 1 -- start start 2024-04-04T21:02:29.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.645+0000 7fe3f1667640 1 librados: setting wanted keys 2024-04-04T21:02:29.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.645+0000 7fe3f1667640 1 librados: calling monclient init 2024-04-04T21:02:29.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.645+0000 7fe3f1667640 10 monclient: init 2024-04-04T21:02:29.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:02:29.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3ec0a9a50 0x7fe3ec135940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 10 monclient(hunting): picked mon.noname-a con 0x7fe3ec0a9a50 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:29.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3ec13f6a0 0x7fe3ec1460c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 10 monclient(hunting): picked mon.noname-c con 0x7fe3ec13f6a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:29.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3ec142880 0x7fe3ec146600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 10 monclient(hunting): picked mon.noname-b con 0x7fe3ec142880 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:29.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec14d070 con 0x7fe3ec0a9a50 2024-04-04T21:02:29.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec14cef0 con 0x7fe3ec142880 2024-04-04T21:02:29.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec14d1f0 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.646+0000 7fe3f1667640 10 monclient(hunting): authenticate will time out at 1398.913973s 2024-04-04T21:02:29.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.647+0000 7fe3ea7fc640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3ec13f6a0 0x7fe3ec1460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.647+0000 7fe3ea7fc640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3ec13f6a0 0x7fe3ec1460c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:33006/0 (socket says 172.21.15.106:33006) 2024-04-04T21:02:29.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.647+0000 7fe3ea7fc640 1 -- 172.21.15.106:0/3731565926 learned_addr learned my addr 172.21.15.106:0/3731565926 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:29.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.647+0000 7fe3ea7fc640 10 monclient(hunting): get_auth_request con 0x7fe3ec13f6a0 auth_method 0 2024-04-04T21:02:29.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.647+0000 7fe3ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.647+0000 7fe3ea7fc640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.647+0000 7fe3ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.648+0000 7fe3ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.648+0000 7fe3ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.648+0000 7fe3ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:29.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.653+0000 7fe3ea7fc640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2024-04-04T21:02:29.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.653+0000 7fe3ea7fc640 1 -- 172.21.15.106:0/3731565926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3ec142880 msgr2=0x7fe3ec146600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.653+0000 7fe3ea7fc640 1 --2- 172.21.15.106:0/3731565926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3ec142880 0x7fe3ec146600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.653+0000 7fe3ea7fc640 1 -- 172.21.15.106:0/3731565926 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3ec0a9a50 msgr2=0x7fe3ec135940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.653+0000 7fe3ea7fc640 1 --2- 172.21.15.106:0/3731565926 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3ec0a9a50 0x7fe3ec135940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.653+0000 7fe3ea7fc640 10 monclient: _finish_hunting 0 2024-04-04T21:02:29.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.653+0000 7fe3ea7fc640 1 monclient: found mon.noname-c 2024-04-04T21:02:29.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.653+0000 7fe3ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:29.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.653+0000 7fe3ea7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:02:29.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.653+0000 7fe3ea7fc640 1 -- 172.21.15.106:0/3731565926 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec13cf80 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.657 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.657+0000 7fe3ea7fc640 1 --2- 172.21.15.106:0/3731565926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3ec13f6a0 0x7fe3ec1460c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0000c00 tx=0x7fe3e00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.657 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.657+0000 7fe3d37fe640 10 client.?.objecter ms_handle_connect 0x7fe3ec13f6a0 2024-04-04T21:02:29.657 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.657+0000 7fe3d37fe640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:29.662 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3d37fe640 1 -- 172.21.15.106:0/3731565926 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0032860 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.662 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.662 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3d37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:29.662 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3d37fe640 10 monclient: dump: 2024-04-04T21:02:29.662 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.662 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.662 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.662 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3d37fe640 10 monclient: _finish_auth 0 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3d37fe640 10 monclient: _check_auth_tickets 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3d37fe640 1 -- 172.21.15.106:0/3731565926 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e0032a20 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3d37fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3d37fe640 1 -- 172.21.15.106:0/3731565926 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0032e10 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3f1667640 5 monclient: authenticate success, global_id 4319 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3f1667640 10 monclient: _renew_subs 2024-04-04T21:02:29.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3f1667640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:29.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3f1667640 1 -- 172.21.15.106:0/3731565926 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ec142220 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3e97fa640 4 set_mon_vals no callback set 2024-04-04T21:02:29.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:29.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3d37fe640 10 monclient: dump: 2024-04-04T21:02:29.664 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.664 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.664 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.664 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.664 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.664 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.664 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.664 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.664 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.665 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3f1667640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:29.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3f1667640 10 monclient: _renew_subs 2024-04-04T21:02:29.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3f1667640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:29.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3f1667640 1 -- 172.21.15.106:0/3731565926 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ec13bdd0 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.661+0000 7fe3f1667640 1 librados: init done 2024-04-04T21:02:29.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.662+0000 7fe3f1667640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.662+0000 7fe3f1667640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.662+0000 7fe3f1667640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:29.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.662+0000 7fe3f1667640 1 -- 172.21.15.106:0/3731565926 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ec04f050 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.671+0000 7f19da7fc640 1 -- 172.21.15.106:0/1581971477 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f19e0038020 con 0x7f19e40a9a50 2024-04-04T21:02:29.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.671+0000 7f19da7fc640 1 --2- 172.21.15.106:0/1581971477 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f19c806ceb0 0x7f19c806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.671+0000 7f19e9580640 1 --2- 172.21.15.106:0/1581971477 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f19c806ceb0 0x7f19c806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.671+0000 7f19e9580640 10 monclient: get_auth_request con 0x7f19c806ceb0 auth_method 0 2024-04-04T21:02:29.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.672+0000 7f19da7fc640 10 client.4344.objecter ms_handle_connect 0x7f19c806ceb0 2024-04-04T21:02:29.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.672+0000 7f19e9580640 1 --2- 172.21.15.106:0/1581971477 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f19c806ceb0 0x7f19c806f370 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f19d4000f60 tx=0x7f19d40079d0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.685+0000 7f19da7fc640 1 -- 172.21.15.106:0/1581971477 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f19e0078ff0 con 0x7f19e40a9a50 2024-04-04T21:02:29.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.685+0000 7f19da7fc640 10 client.4344.objecter ms_dispatch 0x7f19e4000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:29.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.685+0000 7f19da7fc640 3 client.4344.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:29.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.685+0000 7f19da7fc640 3 client.4344.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:29.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.685+0000 7f19da7fc640 20 client.4344.objecter dump_active .. 0 homeless 2024-04-04T21:02:29.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.685+0000 7f19da7fc640 1 -- 172.21.15.106:0/1581971477 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f19e0055080 con 0x7f19e40a9a50 2024-04-04T21:02:29.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.685+0000 7f19da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.685+0000 7f19da7fc640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:29.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-04T21:02:29.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-04T21:02:29.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.706+0000 7f4218aa3640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f421413f810 0x7f421413fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:49576/0 (socket says 172.21.15.106:49576) 2024-04-04T21:02:29.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.706+0000 7f4218aa3640 1 -- 172.21.15.106:0/721684409 learned_addr learned my addr 172.21.15.106:0/721684409 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:29.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.711+0000 7f4212ffd640 1 -- 172.21.15.106:0/721684409 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735641796 0 0) 0x7f42140b2e00 con 0x7f421413f810 2024-04-04T21:02:29.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.711+0000 7f4212ffd640 10 monclient(hunting): _negotiate 2024-04-04T21:02:29.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.711+0000 7f4212ffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.711+0000 7f4212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.711+0000 7f4212ffd640 10 monclient(hunting): my global_id is 4347 2024-04-04T21:02:29.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.711+0000 7f4212ffd640 1 -- 172.21.15.106:0/721684409 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41fc003520 con 0x7f421413f810 2024-04-04T21:02:29.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 -- 172.21.15.106:0/721684409 <== mon.0 v1:172.21.15.106:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1457361460 0 0) 0x7f41fc003520 con 0x7f421413f810 2024-04-04T21:02:29.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 -- 172.21.15.106:0/721684409 >> v2:172.21.15.203:3300/0 conn(0x7f42141426f0 msgr2=0x7f42140b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:29.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 --2- 172.21.15.106:0/721684409 >> v2:172.21.15.203:3300/0 conn(0x7f42141426f0 0x7f42140b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 -- 172.21.15.106:0/721684409 >> v1:172.21.15.203:6789/0 conn(0x7f42140a9b60 legacy=0x7f42141430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:29.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient: _finish_hunting 0 2024-04-04T21:02:29.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 monclient: found mon.noname-a-legacy 2024-04-04T21:02:29.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:29.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.106:6789/0 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 -- 172.21.15.106:0/721684409 --> v1:172.21.15.106:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42140b23b0 con 0x7f421413f810 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient: _finish_auth 0 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 -- 172.21.15.106:0/721684409 <== mon.0 v1:172.21.15.106:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 141466616 0 0) 0x7f420c0019f0 con 0x7f421413f810 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f421a506640 5 monclient: authenticate success, global_id 4347 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f421a506640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient: dump: 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.717 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.718 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.718 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.718 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.718 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] but i'm connected to v1:172.21.15.106:6789/0, reconnecting 2024-04-04T21:02:29.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 -- 172.21.15.106:0/721684409 >> v1:172.21.15.106:6789/0 conn(0x7f421413f810 legacy=0x7f421413fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:02:29.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 --2- 172.21.15.106:0/721684409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41fc003a00 0x7f41fc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient(hunting): picked mon.a con 0x7f41fc003a00 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:29.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 --2- 172.21.15.106:0/721684409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc004800 0x7f41fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient(hunting): picked mon.c con 0x7f41fc004800 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:29.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 --2- 172.21.15.106:0/721684409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41fc005370 0x7f41fc00b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient(hunting): picked mon.b con 0x7f41fc005370 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:29.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 -- 172.21.15.106:0/721684409 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42140a9f40 con 0x7f41fc003a00 2024-04-04T21:02:29.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 -- 172.21.15.106:0/721684409 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc00c2c0 con 0x7f41fc005370 2024-04-04T21:02:29.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 1 -- 172.21.15.106:0/721684409 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc00c4c0 con 0x7f41fc004800 2024-04-04T21:02:29.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.715+0000 7f4212ffd640 10 monclient(hunting): _finish_auth 0 2024-04-04T21:02:29.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.716+0000 7f421a506640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:29.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.720+0000 7f4213fff640 1 --2- 172.21.15.106:0/721684409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41fc003a00 0x7f41fc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.721+0000 7f4213fff640 10 monclient(hunting): get_auth_request con 0x7f41fc003a00 auth_method 0 2024-04-04T21:02:29.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.721+0000 7f4213fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.721+0000 7f4213fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.721+0000 7f4213fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-04T21:02:29.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.722+0000 7f4213fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.722+0000 7f4213fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.722+0000 7f4213fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-04T21:02:29.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.724+0000 7f4213fff640 10 monclient(hunting): handle_auth_done global_id 4347 payload 293 2024-04-04T21:02:29.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.724+0000 7f4213fff640 1 -- 172.21.15.106:0/721684409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc004800 msgr2=0x7f41fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:29.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.724+0000 7f4213fff640 1 --2- 172.21.15.106:0/721684409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc004800 0x7f41fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.724+0000 7f4213fff640 1 -- 172.21.15.106:0/721684409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41fc005370 msgr2=0x7f41fc00b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:29.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.724+0000 7f4213fff640 1 --2- 172.21.15.106:0/721684409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41fc005370 0x7f41fc00b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.724+0000 7f4218aa3640 1 -- 172.21.15.106:0/721684409 reap_dead start 2024-04-04T21:02:29.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.724+0000 7f4213fff640 10 monclient: _finish_hunting 0 2024-04-04T21:02:29.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.724+0000 7f4213fff640 1 monclient: found mon.a 2024-04-04T21:02:29.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.724+0000 7f4213fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:29.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.724+0000 7f4213fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:29.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.724+0000 7f4213fff640 1 -- 172.21.15.106:0/721684409 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc003520 con 0x7f41fc003a00 2024-04-04T21:02:29.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.724+0000 7f4213fff640 10 monclient: _finish_auth 0 2024-04-04T21:02:29.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.724+0000 7f4213fff640 10 monclient: _check_auth_tickets 2024-04-04T21:02:29.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.724+0000 7f4213fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:29.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.727+0000 7f4213fff640 1 --2- 172.21.15.106:0/721684409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41fc003a00 0x7f41fc003e00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f41fc005750 tx=0x7f41fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf82a60fd6ea1a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.731+0000 7f4212ffd640 1 -- 172.21.15.106:0/721684409 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204001220 con 0x7f41fc003a00 2024-04-04T21:02:29.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.731+0000 7f4212ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.731+0000 7f4212ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:29.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.731+0000 7f4212ffd640 10 monclient: dump: 2024-04-04T21:02:29.732 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.732 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.732 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.732 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.732 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.732 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.732 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.732 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.732 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.732 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.731+0000 7f4212ffd640 1 -- 172.21.15.106:0/721684409 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4204005290 con 0x7f41fc003a00 2024-04-04T21:02:29.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.731+0000 7f4212ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:29.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.731+0000 7f4212ffd640 1 -- 172.21.15.106:0/721684409 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204003850 con 0x7f41fc003a00 2024-04-04T21:02:29.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.731+0000 7f421a506640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:29.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.731+0000 7f421a506640 4 set_mon_vals no callback set 2024-04-04T21:02:29.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.731+0000 7f4212ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.731+0000 7f4212ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:29.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.731+0000 7f4212ffd640 10 monclient: dump: 2024-04-04T21:02:29.733 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.733 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.733 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.733 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.733 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.733 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.733 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.733 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.734 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.734 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.734 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.732+0000 7f421a506640 10 monclient: shutdown 2024-04-04T21:02:29.734 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.732+0000 7f421a506640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:29.734 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.732+0000 7f421a506640 1 -- 172.21.15.106:0/721684409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41fc003a00 msgr2=0x7f41fc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.734 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.732+0000 7f421a506640 1 --2- 172.21.15.106:0/721684409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41fc003a00 0x7f41fc003e00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f41fc005750 tx=0x7f41fc005830 comp rx=0 tx=0).stop 2024-04-04T21:02:29.735 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.733+0000 7f421a506640 1 -- 172.21.15.106:0/721684409 shutdown_connections 2024-04-04T21:02:29.735 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.733+0000 7f421a506640 1 --2- 172.21.15.106:0/721684409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41fc003a00 0x7f41fc003e00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.735 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.733+0000 7f421a506640 1 -- 172.21.15.106:0/721684409 >> 172.21.15.106:0/721684409 conn(0x7f42140b7ec0 msgr2=0x7f42140b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:29.735 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.733+0000 7f421a506640 1 -- 172.21.15.106:0/721684409 shutdown_connections 2024-04-04T21:02:29.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.733+0000 7f421a506640 1 -- 172.21.15.106:0/721684409 wait complete. 2024-04-04T21:02:29.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.734+0000 7f421a506640 10 monclient: build_initial_monmap 2024-04-04T21:02:29.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.734+0000 7f421a506640 10 monclient: monmap: 2024-04-04T21:02:29.736 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:29.736 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:29.736 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:29.736 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:29.736 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:29.736 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.736 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:29.736 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:29.736 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:29.736 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.735+0000 7f421a506640 1 librados: starting msgr at 2024-04-04T21:02:29.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.735+0000 7f421a506640 1 librados: starting objecter 2024-04-04T21:02:29.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.735+0000 7f421a506640 1 Processor -- start 2024-04-04T21:02:29.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.736+0000 7f421a506640 1 -- start start 2024-04-04T21:02:29.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.736+0000 7f421a506640 1 librados: setting wanted keys 2024-04-04T21:02:29.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.736+0000 7f421a506640 1 librados: calling monclient init 2024-04-04T21:02:29.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.736+0000 7f421a506640 10 monclient: init 2024-04-04T21:02:29.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.737+0000 7f421a506640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.737+0000 7f421a506640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:02:29.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.737+0000 7f421a506640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc003a00 0x7f42141db3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.737+0000 7f421a506640 10 monclient(hunting): picked mon.noname-c con 0x7f41fc003a00 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.737+0000 7f421a506640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42141db910 0x7f42140b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.737+0000 7f421a506640 10 monclient(hunting): picked mon.noname-b con 0x7f42141db910 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.737+0000 7f421a506640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42140b0fa0 0x7f42141d7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.737+0000 7f421a506640 10 monclient(hunting): picked mon.noname-a con 0x7f42140b0fa0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.737+0000 7f421a506640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.738+0000 7f421a506640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4214142c10 con 0x7f42140b0fa0 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.738+0000 7f421a506640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.738+0000 7f421a506640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42140b2b10 con 0x7f42141db910 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.738+0000 7f421a506640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.738+0000 7f421a506640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42141430a0 con 0x7f41fc003a00 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.738+0000 7f421a506640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.738+0000 7f421a506640 10 monclient(hunting): authenticate will time out at 1399.005554s 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.738+0000 7f4218aa3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42140b0fa0 0x7f42141d7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.739+0000 7f4218aa3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42140b0fa0 0x7f42141d7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35068/0 (socket says 172.21.15.106:35068) 2024-04-04T21:02:29.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.739+0000 7f4218aa3640 1 -- 172.21.15.106:0/1877687748 learned_addr learned my addr 172.21.15.106:0/1877687748 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:29.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.739+0000 7f4218aa3640 10 monclient(hunting): get_auth_request con 0x7f42140b0fa0 auth_method 0 2024-04-04T21:02:29.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.739+0000 7f4218aa3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.739+0000 7f4218aa3640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.739+0000 7f4218aa3640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.740+0000 7f4218aa3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.740+0000 7f4218aa3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.740+0000 7f4218aa3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:29.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.744+0000 7f4218aa3640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-04-04T21:02:29.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.744+0000 7f4218aa3640 1 -- 172.21.15.106:0/1877687748 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc003a00 msgr2=0x7f42141db3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.744+0000 7f4218aa3640 1 --2- 172.21.15.106:0/1877687748 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc003a00 0x7f42141db3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.744+0000 7f4218aa3640 1 -- 172.21.15.106:0/1877687748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42141db910 msgr2=0x7f42140b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.744+0000 7f4218aa3640 1 --2- 172.21.15.106:0/1877687748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42141db910 0x7f42140b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.744+0000 7f4218aa3640 10 monclient: _finish_hunting 0 2024-04-04T21:02:29.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.744+0000 7f4218aa3640 1 monclient: found mon.noname-a 2024-04-04T21:02:29.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.744+0000 7f4218aa3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:29.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.744+0000 7f4218aa3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:29.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.744+0000 7f4218aa3640 1 -- 172.21.15.106:0/1877687748 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42141dbd50 con 0x7f42140b0fa0 2024-04-04T21:02:29.747 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.747+0000 7f4218aa3640 1 --2- 172.21.15.106:0/1877687748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42140b0fa0 0x7f42141d7c60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f42140b48a0 tx=0x7f420c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.747 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.747+0000 7f4210ff9640 10 client.?.objecter ms_handle_connect 0x7f42140b0fa0 2024-04-04T21:02:29.747 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.747+0000 7f4210ff9640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:29.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.753+0000 7f4210ff9640 1 -- 172.21.15.106:0/1877687748 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c00a610 con 0x7f42140b0fa0 2024-04-04T21:02:29.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.753+0000 7f4210ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.753+0000 7f4210ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:29.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.753+0000 7f4210ff9640 10 monclient: dump: 2024-04-04T21:02:29.756 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.756 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.756 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.756 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.756 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.756 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.756 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.756 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.756 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.756 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.753+0000 7f4210ff9640 10 monclient: _finish_auth 0 2024-04-04T21:02:29.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.753+0000 7f4210ff9640 10 monclient: _check_auth_tickets 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.753+0000 7f4210ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.753+0000 7f4210ff9640 1 -- 172.21.15.106:0/1877687748 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420c00a7b0 con 0x7f42140b0fa0 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.753+0000 7f4210ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.753+0000 7f4210ff9640 1 -- 172.21.15.106:0/1877687748 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c00aba0 con 0x7f42140b0fa0 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.753+0000 7f4210ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.753+0000 7f4210ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.753+0000 7f4210ff9640 10 monclient: dump: 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.757 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.758 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.758 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.753+0000 7f42127fc640 4 set_mon_vals no callback set 2024-04-04T21:02:29.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.754+0000 7f421a506640 5 monclient: authenticate success, global_id 4350 2024-04-04T21:02:29.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.754+0000 7f421a506640 10 monclient: _renew_subs 2024-04-04T21:02:29.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.754+0000 7f421a506640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:29.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.754+0000 7f421a506640 1 -- 172.21.15.106:0/1877687748 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4214120400 con 0x7f42140b0fa0 2024-04-04T21:02:29.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.754+0000 7f421a506640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:29.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.754+0000 7f421a506640 10 monclient: _renew_subs 2024-04-04T21:02:29.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.754+0000 7f421a506640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:29.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.754+0000 7f421a506640 1 -- 172.21.15.106:0/1877687748 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42140b1400 con 0x7f42140b0fa0 2024-04-04T21:02:29.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.754+0000 7f421a506640 1 librados: init done 2024-04-04T21:02:29.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.756+0000 7f421a506640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.756+0000 7f421a506640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.756+0000 7f421a506640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:29.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.756+0000 7f421a506640 1 -- 172.21.15.106:0/1877687748 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421404f050 con 0x7f42140b0fa0 2024-04-04T21:02:29.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.805+0000 7f3da1182640 10 monclient: get_monmap_and_config 2024-04-04T21:02:29.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.805+0000 7f3da1182640 10 monclient: build_initial_monmap 2024-04-04T21:02:29.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.805+0000 7f3da1182640 10 monclient: monmap: 2024-04-04T21:02:29.808 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:29.809 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.809 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:29.806879+0000 2024-04-04T21:02:29.809 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:29.806879+0000 2024-04-04T21:02:29.809 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:29.809 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.809 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:29.809 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:29.809 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:29.809 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:29.809 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:29.810 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.810 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.806+0000 7f3da1182640 1 Processor -- start 2024-04-04T21:02:29.810 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.806+0000 7f3da1182640 1 -- start start 2024-04-04T21:02:29.810 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.806+0000 7f3da1182640 10 monclient: init 2024-04-04T21:02:29.810 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.807+0000 7f3da1182640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.810 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.807+0000 7f3da1182640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-04-04T21:02:29.810 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.807+0000 7f3da1182640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d9c0a9a50 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:29.810 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.807+0000 7f3da1182640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3d9c0a9e30 0x7f3d9c142880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.810 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.807+0000 7f3da1182640 10 monclient(hunting): picked mon.noname-a con 0x7f3d9c0a9e30 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:29.810 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.807+0000 7f3da1182640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3d9c142dc0 0x7f3d9c14cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.810 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.807+0000 7f3da1182640 10 monclient(hunting): picked mon.noname-c con 0x7f3d9c142dc0 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:29.810 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.807+0000 7f3da1182640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c0ab750 con 0x7f3d9c0a9a50 2024-04-04T21:02:29.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.807+0000 7f3da1182640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.807+0000 7f3da1182640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9c140120 con 0x7f3d9c0a9e30 2024-04-04T21:02:29.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.807+0000 7f3da1182640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.807+0000 7f3da1182640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9c0aa660 con 0x7f3d9c142dc0 2024-04-04T21:02:29.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.807+0000 7f3da1182640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.807+0000 7f3da1182640 10 monclient(hunting): authenticate will time out at 1399.074584s 2024-04-04T21:02:29.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.818+0000 7f5fd6ffd640 1 -- 172.21.15.106:0/3900206976 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5fe0078dd0 con 0x7f5fe81d7fa0 2024-04-04T21:02:29.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.818+0000 7f5fd6ffd640 10 client.4390.objecter ms_dispatch 0x7f5fe8000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:29.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.818+0000 7f5fd6ffd640 3 client.4390.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:29.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.818+0000 7f5fd6ffd640 3 client.4390.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:29.819 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.818+0000 7f5fd6ffd640 20 client.4390.objecter dump_active .. 0 homeless 2024-04-04T21:02:29.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.829+0000 7f5fd6ffd640 1 -- 172.21.15.106:0/3900206976 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5fe004a060 con 0x7f5fe81d7fa0 2024-04-04T21:02:29.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.829+0000 7f5fd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.829+0000 7f5fd6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:29.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.829+0000 7ff858a78640 10 monclient: get_monmap_and_config 2024-04-04T21:02:29.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.829+0000 7ff858a78640 10 monclient: build_initial_monmap 2024-04-04T21:02:29.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.829+0000 7ff858a78640 10 monclient: monmap: 2024-04-04T21:02:29.830 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:29.830 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.830 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:29.830368+0000 2024-04-04T21:02:29.830 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:29.830368+0000 2024-04-04T21:02:29.830 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:29.830 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.830 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:29.830 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:29.830 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:29.830 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:29.830 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:29.831 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.830+0000 7ff858a78640 1 Processor -- start 2024-04-04T21:02:29.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.830+0000 7ff858a78640 1 -- start start 2024-04-04T21:02:29.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.830+0000 7ff858a78640 10 monclient: init 2024-04-04T21:02:29.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.831+0000 7ff858a78640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.831+0000 7ff858a78640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-04-04T21:02:29.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.831+0000 7ff858a78640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8540a9a50 0x7ff8540a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.831+0000 7ff858a78640 10 monclient(hunting): picked mon.noname-b con 0x7ff8540a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:29.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.831+0000 7ff858a78640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff85413f6a0 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:29.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.831+0000 7ff858a78640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff85413fac0 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:29.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.831+0000 7ff858a78640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8540ab750 con 0x7ff85413f6a0 2024-04-04T21:02:29.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.831+0000 7ff858a78640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff854140120 con 0x7ff85413fac0 2024-04-04T21:02:29.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.831+0000 7ff858a78640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.831+0000 7ff858a78640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85413ff50 con 0x7ff8540a9a50 2024-04-04T21:02:29.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.831+0000 7ff858a78640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.831+0000 7ff858a78640 10 monclient(hunting): authenticate will time out at 1399.098512s 2024-04-04T21:02:29.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.853+0000 7f3d9b59e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3d9c142dc0 0x7f3d9c14cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.854+0000 7f3d9ad9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f3d9c0a9a50 0x7f3d9c13f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:41944/0 (socket says 172.21.15.106:41944) 2024-04-04T21:02:29.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.854+0000 7f3d9ad9d640 1 -- 172.21.15.106:0/712195376 learned_addr learned my addr 172.21.15.106:0/712195376 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:29.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.854+0000 7f3d9b59e640 10 monclient(hunting): get_auth_request con 0x7f3d9c142dc0 auth_method 0 2024-04-04T21:02:29.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.854+0000 7f3d9b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.854+0000 7f3d9b59e640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.855 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.854+0000 7f3d9b59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.859+0000 7f3d9b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.859+0000 7f3d9b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.859+0000 7f3d9b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:29.861 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.861+0000 7ff852d9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7ff85413fac0 0x7ff85414cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:41946/0 (socket says 172.21.15.106:41946) 2024-04-04T21:02:29.861 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.861+0000 7ff852d9d640 1 -- 172.21.15.106:0/2666941334 learned_addr learned my addr 172.21.15.106:0/2666941334 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:29.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.862+0000 7f3d9b59e640 10 monclient(hunting): handle_auth_done global_id 4393 payload 274 2024-04-04T21:02:29.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.862+0000 7f3d9b59e640 1 -- 172.21.15.106:0/712195376 >> v2:172.21.15.106:3300/0 conn(0x7f3d9c0a9e30 msgr2=0x7f3d9c142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:29.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.862+0000 7f3d9b59e640 1 --2- 172.21.15.106:0/712195376 >> v2:172.21.15.106:3300/0 conn(0x7f3d9c0a9e30 0x7f3d9c142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.862+0000 7f3d9b59e640 1 -- 172.21.15.106:0/712195376 >> v1:172.21.15.203:6789/0 conn(0x7f3d9c0a9a50 legacy=0x7f3d9c13f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.862+0000 7f3d9b59e640 10 monclient: _finish_hunting 0 2024-04-04T21:02:29.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.862+0000 7f3d9b59e640 1 monclient: found mon.noname-c 2024-04-04T21:02:29.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.862+0000 7f3d9b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:29.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.862+0000 7f3d9b59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:02:29.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.862+0000 7f3d9b59e640 1 -- 172.21.15.106:0/712195376 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c0aa250 con 0x7f3d9c142dc0 2024-04-04T21:02:29.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.866+0000 7f3d9b59e640 1 --2- 172.21.15.106:0/712195376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c142dc0 0x7f3d9c14cef0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0b4550 tx=0x7f3d9402be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70335f9b38c7af23 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3d99d9b640 1 -- 172.21.15.106:0/712195376 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94039070 con 0x7f3d9c142dc0 2024-04-04T21:02:29.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3d99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3d99d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:29.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3d99d9b640 10 monclient: dump: 2024-04-04T21:02:29.871 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.871 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.871 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.871 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.871 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.871 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.872 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.872 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.872 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.872 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3d99d9b640 10 monclient: _finish_auth 0 2024-04-04T21:02:29.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3d99d9b640 10 monclient: _check_auth_tickets 2024-04-04T21:02:29.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3d99d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:29.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3d99d9b640 1 -- 172.21.15.106:0/712195376 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9402cbf0 con 0x7f3d9c142dc0 2024-04-04T21:02:29.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3d99d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:29.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3d99d9b640 1 -- 172.21.15.106:0/712195376 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94034440 con 0x7f3d9c142dc0 2024-04-04T21:02:29.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3d99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3d99d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:29.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3d99d9b640 10 monclient: dump: 2024-04-04T21:02:29.872 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.873 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.873 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.873 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.873 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.873 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.873 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.873 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.873 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.874 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3da1182640 5 monclient: authenticate success, global_id 4393 2024-04-04T21:02:29.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3da1182640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:29.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.870+0000 7f3da1182640 4 set_mon_vals no callback set 2024-04-04T21:02:29.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.871+0000 7f3da1182640 10 monclient: shutdown 2024-04-04T21:02:29.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.871+0000 7f3da1182640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:29.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.871+0000 7f3da1182640 1 -- 172.21.15.106:0/712195376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c142dc0 msgr2=0x7f3d9c14cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.871+0000 7f3da1182640 1 --2- 172.21.15.106:0/712195376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c142dc0 0x7f3d9c14cef0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0b4550 tx=0x7f3d9402be40 comp rx=0 tx=0).stop 2024-04-04T21:02:29.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.871+0000 7f3da1182640 1 -- 172.21.15.106:0/712195376 shutdown_connections 2024-04-04T21:02:29.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.871+0000 7f3da1182640 1 --2- 172.21.15.106:0/712195376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c142dc0 0x7f3d9c14cef0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.871+0000 7f3da1182640 1 --2- 172.21.15.106:0/712195376 >> v2:172.21.15.106:3300/0 conn(0x7f3d9c0a9e30 0x7f3d9c142880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.871+0000 7f3da1182640 1 -- 172.21.15.106:0/712195376 >> 172.21.15.106:0/712195376 conn(0x7f3d9c0b8030 msgr2=0x7f3d9c0b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:29.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.871+0000 7f3da1182640 1 -- 172.21.15.106:0/712195376 shutdown_connections 2024-04-04T21:02:29.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.871+0000 7f3da1182640 1 -- 172.21.15.106:0/712195376 wait complete. 2024-04-04T21:02:29.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.872+0000 7f3da1182640 10 monclient: build_initial_monmap 2024-04-04T21:02:29.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.872+0000 7f3da1182640 10 monclient: monmap: 2024-04-04T21:02:29.879 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:29.879 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:29.880 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:29.880 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:29.880 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:29.880 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.882 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:29.882 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:29.882 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:29.882 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.872+0000 7f3da1182640 1 librados: starting msgr at 2024-04-04T21:02:29.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.872+0000 7f3da1182640 1 librados: starting objecter 2024-04-04T21:02:29.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.872+0000 7f3da1182640 1 Processor -- start 2024-04-04T21:02:29.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 1 -- start start 2024-04-04T21:02:29.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 1 librados: setting wanted keys 2024-04-04T21:02:29.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 1 librados: calling monclient init 2024-04-04T21:02:29.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 10 monclient: init 2024-04-04T21:02:29.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:29.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-04T21:02:29.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c0a9a50 0x7f3d9c146570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 10 monclient(hunting): picked mon.noname-b con 0x7f3d9c0a9a50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:29.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d9c0a9e30 0x7f3d9c1445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 10 monclient(hunting): picked mon.noname-a con 0x7f3d9c0a9e30 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:29.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d9c142dc0 0x7f3d9c1d7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 10 monclient(hunting): picked mon.noname-c con 0x7f3d9c142dc0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:29.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c1433b0 con 0x7f3d9c0a9e30 2024-04-04T21:02:29.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c143230 con 0x7f3d9c0a9a50 2024-04-04T21:02:29.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:29.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c14d430 con 0x7f3d9c142dc0 2024-04-04T21:02:29.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:29.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.873+0000 7f3da1182640 10 monclient(hunting): authenticate will time out at 1399.140825s 2024-04-04T21:02:29.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.874+0000 7f3d9ad9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c0a9a50 0x7f3d9c146570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.874+0000 7f3d9ad9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c0a9a50 0x7f3d9c146570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50444/0 (socket says 172.21.15.106:50444) 2024-04-04T21:02:29.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.874+0000 7f3d9ad9d640 1 -- 172.21.15.106:0/2655651918 learned_addr learned my addr 172.21.15.106:0/2655651918 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:29.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.874+0000 7f3d9ad9d640 10 monclient(hunting): get_auth_request con 0x7f3d9c0a9a50 auth_method 0 2024-04-04T21:02:29.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.874+0000 7f3d9ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:29.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.874+0000 7f3d9ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:29.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.874+0000 7f3d9ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:29.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.875+0000 7f3d9ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:29.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.875+0000 7f3d9ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:29.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.875+0000 7f3d9ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:29.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.877+0000 7f3d9ad9d640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2024-04-04T21:02:29.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.877+0000 7f3d9ad9d640 1 -- 172.21.15.106:0/2655651918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d9c142dc0 msgr2=0x7f3d9c1d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.877+0000 7f3d9ad9d640 1 --2- 172.21.15.106:0/2655651918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d9c142dc0 0x7f3d9c1d7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.877+0000 7f3d9ad9d640 1 -- 172.21.15.106:0/2655651918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d9c0a9e30 msgr2=0x7f3d9c1445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:29.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.877+0000 7f3d9ad9d640 1 --2- 172.21.15.106:0/2655651918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d9c0a9e30 0x7f3d9c1445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:29.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.877+0000 7f3d9ad9d640 10 monclient: _finish_hunting 0 2024-04-04T21:02:29.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.877+0000 7f3d9ad9d640 1 monclient: found mon.noname-b 2024-04-04T21:02:29.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.877+0000 7f3d9ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:29.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.877+0000 7f3d9ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:29.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.877+0000 7f3d9ad9d640 1 -- 172.21.15.106:0/2655651918 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c143af0 con 0x7f3d9c0a9a50 2024-04-04T21:02:29.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.879+0000 7f3d9ad9d640 1 --2- 172.21.15.106:0/2655651918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c0a9a50 0x7f3d9c146570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c00c4d0 tx=0x7f3d8c00c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.879+0000 7f3d837fe640 10 client.?.objecter ms_handle_connect 0x7f3d9c0a9a50 2024-04-04T21:02:29.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.879+0000 7f3d837fe640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:29.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.881+0000 7fe3d37fe640 1 -- 172.21.15.106:0/3731565926 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe3e004c060 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.882+0000 7fe3d37fe640 1 --2- 172.21.15.106:0/3731565926 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe3d806ceb0 0x7fe3d806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.882+0000 7fe3eaffd640 1 --2- 172.21.15.106:0/3731565926 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe3d806ceb0 0x7fe3d806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.882+0000 7fe3eaffd640 10 monclient: get_auth_request con 0x7fe3d806ceb0 auth_method 0 2024-04-04T21:02:29.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.882+0000 7fe3eaffd640 1 --2- 172.21.15.106:0/3731565926 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe3d806ceb0 0x7fe3d806f370 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc009710 tx=0x7fe3dc006c90 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.882+0000 7fe3d37fe640 10 client.4319.objecter ms_handle_connect 0x7fe3d806ceb0 2024-04-04T21:02:29.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3d837fe640 1 -- 172.21.15.106:0/2655651918 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c017070 con 0x7f3d9c0a9a50 2024-04-04T21:02:29.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3d837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3d837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:29.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3d837fe640 10 monclient: dump: 2024-04-04T21:02:29.891 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.891 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.892 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.892 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.892 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.892 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.892 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.892 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.892 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.892 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3d837fe640 10 monclient: _finish_auth 0 2024-04-04T21:02:29.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3d837fe640 10 monclient: _check_auth_tickets 2024-04-04T21:02:29.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3d837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:29.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3d837fe640 1 -- 172.21.15.106:0/2655651918 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8c005ce0 con 0x7f3d9c0a9a50 2024-04-04T21:02:29.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3d837fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3d837fe640 1 -- 172.21.15.106:0/2655651918 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c007270 con 0x7f3d9c0a9a50 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3d837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3d837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3d837fe640 10 monclient: dump: 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:29.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3da1182640 5 monclient: authenticate success, global_id 4396 2024-04-04T21:02:29.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3d9959a640 4 set_mon_vals no callback set 2024-04-04T21:02:29.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3da1182640 10 monclient: _renew_subs 2024-04-04T21:02:29.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3da1182640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:29.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3da1182640 1 -- 172.21.15.106:0/2655651918 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9c144af0 con 0x7f3d9c0a9a50 2024-04-04T21:02:29.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3da1182640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:29.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3da1182640 10 monclient: _renew_subs 2024-04-04T21:02:29.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3da1182640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:29.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3da1182640 1 -- 172.21.15.106:0/2655651918 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9c1d8450 con 0x7f3d9c0a9a50 2024-04-04T21:02:29.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.883+0000 7f3da1182640 1 librados: init done 2024-04-04T21:02:29.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.884+0000 7f3da1182640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.884+0000 7f3da1182640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:29.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.884+0000 7f3da1182640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:29.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.884+0000 7f3da1182640 1 -- 172.21.15.106:0/2655651918 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9c04f050 con 0x7f3d9c0a9a50 2024-04-04T21:02:29.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.904+0000 7f3d837fe640 1 -- 172.21.15.106:0/2655651918 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3d8c01a070 con 0x7f3d9c0a9a50 2024-04-04T21:02:29.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.904+0000 7f3d837fe640 1 --2- 172.21.15.106:0/2655651918 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f3d8806cee0 0x7f3d8806f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:29.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.904+0000 7f3d9a59c640 1 --2- 172.21.15.106:0/2655651918 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f3d8806cee0 0x7f3d8806f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:29.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.904+0000 7f3d9a59c640 10 monclient: get_auth_request con 0x7f3d8806cee0 auth_method 0 2024-04-04T21:02:29.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.905+0000 7f3d9a59c640 1 --2- 172.21.15.106:0/2655651918 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f3d8806cee0 0x7f3d8806f3a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c13f2c0 tx=0x7f3d90006c90 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:29.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.905+0000 7f3d837fe640 10 client.4396.objecter ms_handle_connect 0x7f3d8806cee0 2024-04-04T21:02:29.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.905+0000 7f3d837fe640 1 -- 172.21.15.106:0/2655651918 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3d8c056210 con 0x7f3d9c0a9a50 2024-04-04T21:02:29.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.905+0000 7f3d837fe640 10 client.4396.objecter ms_dispatch 0x7f3d9c000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:29.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.905+0000 7f3d837fe640 3 client.4396.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:29.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.905+0000 7f3d837fe640 3 client.4396.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:29.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.905+0000 7f3d837fe640 20 client.4396.objecter dump_active .. 0 homeless 2024-04-04T21:02:29.907 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-04T21:02:29.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.917+0000 7fe69c33d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-04T21:02:29.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.917+0000 7fe69c33d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-04T21:02:29.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.917+0000 7fe69c33d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:29.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.917+0000 7fe69c33d640 1 -- 172.21.15.106:0/723836038 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe69413af00 con 0x7fe6940a9e30 2024-04-04T21:02:29.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.919+0000 7fe3d37fe640 1 -- 172.21.15.106:0/3731565926 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe3e0078690 con 0x7fe3ec13f6a0 2024-04-04T21:02:29.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.919+0000 7fe3d37fe640 10 client.4319.objecter ms_dispatch 0x7fe3ec000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:29.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.919+0000 7fe3d37fe640 3 client.4319.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:29.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.919+0000 7fe3d37fe640 3 client.4319.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:29.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.919+0000 7fe3d37fe640 20 client.4319.objecter dump_active .. 0 homeless 2024-04-04T21:02:29.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:29.919+0000 7ff85259c640 1 --2- 172.21.15.106:0/2666941334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8540a9a50 0x7ff8540a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:30.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.009+0000 7f51e1671640 10 monclient: get_monmap_and_config 2024-04-04T21:02:30.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.009+0000 7f51e1671640 10 monclient: build_initial_monmap 2024-04-04T21:02:30.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.009+0000 7f51e1671640 10 monclient: monmap: 2024-04-04T21:02:30.011 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:30.011 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:30.011 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:30.010671+0000 2024-04-04T21:02:30.011 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:30.010671+0000 2024-04-04T21:02:30.011 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:30.011 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:30.011 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:30.011 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:30.011 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:30.011 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:30.011 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:30.011 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:30.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.010+0000 7f51e1671640 1 Processor -- start 2024-04-04T21:02:30.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.010+0000 7f51e1671640 1 -- start start 2024-04-04T21:02:30.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.010+0000 7f51e1671640 10 monclient: init 2024-04-04T21:02:30.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.010+0000 7f51e1671640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:30.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.010+0000 7f51e1671640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-04-04T21:02:30.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.010+0000 7f51e1671640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51dc0a9a50 0x7f51dc0a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:30.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.010+0000 7f51e1671640 10 monclient(hunting): picked mon.noname-b con 0x7f51dc0a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:30.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.010+0000 7f51e1671640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f51dc13f6a0 0x7f51dc13fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:30.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.011+0000 7f51e1671640 10 monclient(hunting): picked mon.noname-c con 0x7f51dc13f6a0 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:30.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.011+0000 7f51e1671640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f51dc142880 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:30.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.011+0000 7f51e1671640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51dc143420 con 0x7f51dc142880 2024-04-04T21:02:30.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.011+0000 7f51e1671640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:30.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.011+0000 7f51e1671640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51dc0aa370 con 0x7f51dc13f6a0 2024-04-04T21:02:30.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.011+0000 7f51e1671640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:30.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.011+0000 7f51e1671640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc0ab750 con 0x7f51dc0a9a50 2024-04-04T21:02:30.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.011+0000 7f51e1671640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:30.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.011+0000 7f51e1671640 10 monclient(hunting): authenticate will time out at 1399.278350s 2024-04-04T21:02:30.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.013+0000 7ff85259c640 10 monclient(hunting): get_auth_request con 0x7ff8540a9a50 auth_method 0 2024-04-04T21:02:30.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.013+0000 7ff85259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:30.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.013+0000 7ff85259c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:30.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.013+0000 7ff85259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:30.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.016+0000 7ff85259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:30.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.016+0000 7ff85259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:30.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.016+0000 7ff85259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:30.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.019+0000 7ff85259c640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-04-04T21:02:30.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.019+0000 7ff85259c640 1 -- 172.21.15.106:0/2666941334 >> v1:172.21.15.203:6789/0 conn(0x7ff85413fac0 legacy=0x7ff85414cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.019+0000 7ff85259c640 1 -- 172.21.15.106:0/2666941334 >> v1:172.21.15.106:6789/0 conn(0x7ff85413f6a0 legacy=0x7ff854142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:30.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.019+0000 7ff85259c640 10 monclient: _finish_hunting 0 2024-04-04T21:02:30.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.019+0000 7ff85259c640 1 monclient: found mon.noname-b 2024-04-04T21:02:30.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.019+0000 7ff85259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:30.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.019+0000 7ff85259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:30.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.019+0000 7ff85259c640 1 -- 172.21.15.106:0/2666941334 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8540aa660 con 0x7ff8540a9a50 2024-04-04T21:02:30.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.023+0000 7fe68affd640 1 -- 172.21.15.106:0/723836038 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe68407a9d0 con 0x7fe6940a9e30 2024-04-04T21:02:30.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.023+0000 7fe68affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-04T21:02:30.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.023+0000 7fe68affd640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:30.024 INFO:teuthology.orchestra.run.smithi106.stdout:64424509445 2024-04-04T21:02:30.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.023+0000 7f51daffd640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51dc0a9a50 0x7f51dc0a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:30.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.024+0000 7ff85259c640 1 --2- 172.21.15.106:0/2666941334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8540a9a50 0x7ff8540a9e30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff8440097d0 tx=0x7ff84402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f726b8fa41e2989 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:30.026 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 10 librados: watch_flush enter 2024-04-04T21:02:30.026 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 10 librados: watch_flush exit 2024-04-04T21:02:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 20 client.4313.objecter shutdown clearing up homeless session... 2024-04-04T21:02:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 10 client.4313.objecter successfully canceled tick 2024-04-04T21:02:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 1 -- 172.21.15.106:0/723836038 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe67806ce60 msgr2=0x7fe67806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 1 --2- 172.21.15.106:0/723836038 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe67806ce60 0x7fe67806f320 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe690000f60 tx=0x7fe690009040 comp rx=0 tx=0).stop 2024-04-04T21:02:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 10 monclient: shutdown 2024-04-04T21:02:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 1 -- 172.21.15.106:0/723836038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6940a9e30 msgr2=0x7fe6941d79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 1 --2- 172.21.15.106:0/723836038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6940a9e30 0x7fe6941d79c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe6840098c0 tx=0x7fe6840046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 1 -- 172.21.15.106:0/723836038 shutdown_connections 2024-04-04T21:02:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 1 --2- 172.21.15.106:0/723836038 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe67806ce60 0x7fe67806f320 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 1 --2- 172.21.15.106:0/723836038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe694142dc0 0x7fe6941d7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 1 --2- 172.21.15.106:0/723836038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6940a9e30 0x7fe6941d79c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 1 --2- 172.21.15.106:0/723836038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6940a9a50 0x7fe694138aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 1 -- 172.21.15.106:0/723836038 >> 172.21.15.106:0/723836038 conn(0x7fe6940b8030 msgr2=0x7fe6940b8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:30.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 1 -- 172.21.15.106:0/723836038 shutdown_connections 2024-04-04T21:02:30.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.025+0000 7fe69c33d640 1 -- 172.21.15.106:0/723836038 wait complete. 2024-04-04T21:02:30.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.026+0000 7fe69c33d640 1 librados: shutdown 2024-04-04T21:02:30.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.026+0000 7f51daffd640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51dc0a9a50 0x7f51dc0a9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:33052/0 (socket says 172.21.15.106:33052) 2024-04-04T21:02:30.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.026+0000 7f51daffd640 1 -- 172.21.15.106:0/1936636969 learned_addr learned my addr 172.21.15.106:0/1936636969 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:30.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.026+0000 7f51daffd640 10 monclient(hunting): get_auth_request con 0x7f51dc0a9a50 auth_method 0 2024-04-04T21:02:30.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.026+0000 7f51daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:30.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.026+0000 7f51daffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:30.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.026+0000 7f51daffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:30.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.028+0000 7f51daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:30.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.028+0000 7f51daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:30.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.028+0000 7f51daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:30.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.032+0000 7f51daffd640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-04-04T21:02:30.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.032+0000 7f51daffd640 1 -- 172.21.15.106:0/1936636969 >> v2:172.21.15.203:3300/0 conn(0x7f51dc13f6a0 msgr2=0x7f51dc13fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:30.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.032+0000 7f51daffd640 1 --2- 172.21.15.106:0/1936636969 >> v2:172.21.15.203:3300/0 conn(0x7f51dc13f6a0 0x7f51dc13fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.032+0000 7f51daffd640 1 -- 172.21.15.106:0/1936636969 >> v1:172.21.15.203:6789/0 conn(0x7f51dc142880 legacy=0x7f51dc142d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:30.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.032+0000 7f51daffd640 10 monclient: _finish_hunting 0 2024-04-04T21:02:30.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.032+0000 7f51daffd640 1 monclient: found mon.noname-b 2024-04-04T21:02:30.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.032+0000 7f51daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:30.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.032+0000 7f51daffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:30.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.032+0000 7f51daffd640 1 -- 172.21.15.106:0/1936636969 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc140120 con 0x7f51dc0a9a50 2024-04-04T21:02:30.041 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.0 2024-04-04T21:02:30.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.044+0000 7fe3d37fe640 1 -- 172.21.15.106:0/3731565926 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe3e0040070 con 0x7fe3ec13f6a0 2024-04-04T21:02:30.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.044+0000 7fe3d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:30.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.044+0000 7fe3d37fe640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:30.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.047+0000 7ff85159a640 1 -- 172.21.15.106:0/2666941334 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844031770 con 0x7ff8540a9a50 2024-04-04T21:02:30.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.047+0000 7ff85159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:30.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff85159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff85159a640 10 monclient: dump: 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff85159a640 10 monclient: _finish_auth 0 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff85159a640 10 monclient: _check_auth_tickets 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff85159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:30.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff85159a640 1 -- 172.21.15.106:0/2666941334 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff844037070 con 0x7ff8540a9a50 2024-04-04T21:02:30.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff85159a640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:30.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff85159a640 1 -- 172.21.15.106:0/2666941334 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844031b30 con 0x7ff8540a9a50 2024-04-04T21:02:30.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff85159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:30.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff85159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:30.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff85159a640 10 monclient: dump: 2024-04-04T21:02:30.050 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:30.050 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:30.050 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.050 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.050 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:30.050 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:30.050 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:30.050 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:30.050 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:30.051 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:30.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff858a78640 5 monclient: authenticate success, global_id 4322 2024-04-04T21:02:30.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff858a78640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:30.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff858a78640 4 set_mon_vals no callback set 2024-04-04T21:02:30.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7f51daffd640 1 --2- 172.21.15.106:0/1936636969 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51dc0a9a50 0x7f51dc0a9e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009870 tx=0x7f51cc02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53fb87827d0e4ff7 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:30.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff858a78640 10 monclient: shutdown 2024-04-04T21:02:30.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff858a78640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:30.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.048+0000 7ff858a78640 1 -- 172.21.15.106:0/2666941334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8540a9a50 msgr2=0x7ff8540a9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.049+0000 7ff858a78640 1 --2- 172.21.15.106:0/2666941334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8540a9a50 0x7ff8540a9e30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff8440097d0 tx=0x7ff84402b810 comp rx=0 tx=0).stop 2024-04-04T21:02:30.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.049+0000 7ff858a78640 1 -- 172.21.15.106:0/2666941334 shutdown_connections 2024-04-04T21:02:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.049+0000 7ff858a78640 1 --2- 172.21.15.106:0/2666941334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8540a9a50 0x7ff8540a9e30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.049+0000 7ff858a78640 1 -- 172.21.15.106:0/2666941334 >> 172.21.15.106:0/2666941334 conn(0x7ff8540b8030 msgr2=0x7ff8540b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.049+0000 7ff858a78640 1 -- 172.21.15.106:0/2666941334 shutdown_connections 2024-04-04T21:02:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.049+0000 7ff858a78640 1 -- 172.21.15.106:0/2666941334 wait complete. 2024-04-04T21:02:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.050+0000 7ff858a78640 10 monclient: build_initial_monmap 2024-04-04T21:02:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.050+0000 7ff858a78640 10 monclient: monmap: 2024-04-04T21:02:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.050+0000 7ff858a78640 1 librados: starting msgr at 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.050+0000 7ff858a78640 1 librados: starting objecter 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.050+0000 7ff858a78640 1 Processor -- start 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.050+0000 7ff858a78640 1 -- start start 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.050+0000 7ff858a78640 1 librados: setting wanted keys 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.050+0000 7ff858a78640 1 librados: calling monclient init 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.050+0000 7ff858a78640 10 monclient: init 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8540a9a50 0x7ff8541d8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 10 monclient(hunting): picked mon.noname-b con 0x7ff8540a9a50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff85413f6a0 0x7ff8541d87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:30.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 10 monclient(hunting): picked mon.noname-c con 0x7ff85413f6a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:30.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff85413fac0 0x7ff8540ad590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:30.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 10 monclient(hunting): picked mon.noname-a con 0x7ff85413fac0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:30.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:30.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8541433e0 con 0x7ff85413fac0 2024-04-04T21:02:30.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:30.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff854143260 con 0x7ff8540a9a50 2024-04-04T21:02:30.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:30.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8540aa370 con 0x7ff85413f6a0 2024-04-04T21:02:30.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:30.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.051+0000 7ff858a78640 10 monclient(hunting): authenticate will time out at 1399.318482s 2024-04-04T21:02:30.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7ff851d9b640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff85413f6a0 0x7ff8541d87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:30.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51d9ffb640 1 -- 172.21.15.106:0/1936636969 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc0317c0 con 0x7f51dc0a9a50 2024-04-04T21:02:30.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:30.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51d9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:30.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51d9ffb640 10 monclient: dump: 2024-04-04T21:02:30.058 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:30.058 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:30.058 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.058 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.058 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:30.058 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:30.058 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:30.058 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51d9ffb640 10 monclient: _finish_auth 0 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51d9ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51d9ffb640 1 -- 172.21.15.106:0/1936636969 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc037070 con 0x7f51dc0a9a50 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51d9ffb640 1 -- 172.21.15.106:0/1936636969 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc031b80 con 0x7f51dc0a9a50 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51d9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51d9ffb640 10 monclient: dump: 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.059 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.060 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:30.060 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:30.060 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:30.060 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:30.060 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:30.060 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:30.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51e1671640 5 monclient: authenticate success, global_id 4325 2024-04-04T21:02:30.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51e1671640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:30.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.054+0000 7f51e1671640 4 set_mon_vals no callback set 2024-04-04T21:02:30.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7ff851d9b640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff85413f6a0 0x7ff8541d87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:33068/0 (socket says 172.21.15.106:33068) 2024-04-04T21:02:30.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7ff851d9b640 1 -- 172.21.15.106:0/3094762490 learned_addr learned my addr 172.21.15.106:0/3094762490 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:30.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7ff851d9b640 10 monclient(hunting): get_auth_request con 0x7ff85413f6a0 auth_method 0 2024-04-04T21:02:30.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7ff851d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:30.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7ff851d9b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:30.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7ff851d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:30.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7f51e1671640 10 monclient: shutdown 2024-04-04T21:02:30.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7f51e1671640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:30.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7f51e1671640 1 -- 172.21.15.106:0/1936636969 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51dc0a9a50 msgr2=0x7f51dc0a9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7f51e1671640 1 --2- 172.21.15.106:0/1936636969 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51dc0a9a50 0x7f51dc0a9e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009870 tx=0x7f51cc02b860 comp rx=0 tx=0).stop 2024-04-04T21:02:30.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7f51e1671640 1 -- 172.21.15.106:0/1936636969 shutdown_connections 2024-04-04T21:02:30.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7f51e1671640 1 --2- 172.21.15.106:0/1936636969 >> v2:172.21.15.203:3300/0 conn(0x7f51dc13f6a0 0x7f51dc13fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7f51e1671640 1 --2- 172.21.15.106:0/1936636969 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51dc0a9a50 0x7f51dc0a9e30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7f51e1671640 1 -- 172.21.15.106:0/1936636969 >> 172.21.15.106:0/1936636969 conn(0x7f51dc0b8030 msgr2=0x7f51dc0b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:30.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7f51e1671640 1 -- 172.21.15.106:0/1936636969 shutdown_connections 2024-04-04T21:02:30.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7f51e1671640 1 -- 172.21.15.106:0/1936636969 wait complete. 2024-04-04T21:02:30.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7ff851d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:30.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7ff851d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:30.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.055+0000 7ff851d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:30.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.056+0000 7f51e1671640 10 monclient: build_initial_monmap 2024-04-04T21:02:30.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.056+0000 7f51e1671640 10 monclient: monmap: 2024-04-04T21:02:30.065 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:30.065 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:30.065 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:30.065 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:30.065 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:30.065 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:30.066 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:30.066 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:30.066 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:30.066 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:30.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.056+0000 7f51e1671640 1 librados: starting msgr at 2024-04-04T21:02:30.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.056+0000 7f51e1671640 1 librados: starting objecter 2024-04-04T21:02:30.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.057+0000 7f51e1671640 1 Processor -- start 2024-04-04T21:02:30.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.057+0000 7f51e1671640 1 -- start start 2024-04-04T21:02:30.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.057+0000 7f51e1671640 1 librados: setting wanted keys 2024-04-04T21:02:30.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.057+0000 7f51e1671640 1 librados: calling monclient init 2024-04-04T21:02:30.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.057+0000 7f51e1671640 10 monclient: init 2024-04-04T21:02:30.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.057+0000 7f51e1671640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:30.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.057+0000 7f51e1671640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:02:30.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.057+0000 7f51e1671640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51dc0a9a50 0x7f51dc146540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:30.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.057+0000 7f51e1671640 10 monclient(hunting): picked mon.noname-c con 0x7f51dc0a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:30.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.057+0000 7f51e1671640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51dc13f6a0 0x7f51dc1399b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:30.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.057+0000 7f51e1671640 10 monclient(hunting): picked mon.noname-b con 0x7f51dc13f6a0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:30.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7f51e1671640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51dc142880 0x7f51dc139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:30.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7f51e1671640 10 monclient(hunting): picked mon.noname-a con 0x7f51dc142880 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:30.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7f51e1671640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:30.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7f51e1671640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc14d070 con 0x7f51dc142880 2024-04-04T21:02:30.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7f51e1671640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:30.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7f51e1671640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc14cef0 con 0x7f51dc13f6a0 2024-04-04T21:02:30.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7f51e1671640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:30.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7f51e1671640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc14d1f0 con 0x7f51dc0a9a50 2024-04-04T21:02:30.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7f51e1671640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:30.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7f51e1671640 10 monclient(hunting): authenticate will time out at 1399.325368s 2024-04-04T21:02:30.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7ff851d9b640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-04-04T21:02:30.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7ff851d9b640 1 -- 172.21.15.106:0/3094762490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8540a9a50 msgr2=0x7ff8541d8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7ff851d9b640 1 --2- 172.21.15.106:0/3094762490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8540a9a50 0x7ff8541d8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7ff851d9b640 1 -- 172.21.15.106:0/3094762490 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff85413fac0 msgr2=0x7ff8540ad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7ff851d9b640 1 --2- 172.21.15.106:0/3094762490 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff85413fac0 0x7ff8540ad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7ff851d9b640 10 monclient: _finish_hunting 0 2024-04-04T21:02:30.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7ff851d9b640 1 monclient: found mon.noname-c 2024-04-04T21:02:30.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7ff851d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:30.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7ff851d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:02:30.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.058+0000 7ff851d9b640 1 -- 172.21.15.106:0/3094762490 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff854144e70 con 0x7ff85413f6a0 2024-04-04T21:02:30.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.062+0000 7ff851d9b640 1 --2- 172.21.15.106:0/3094762490 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff85413f6a0 0x7ff8541d87c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff84800c410 tx=0x7ff84800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:30.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.062+0000 7ff842ffd640 10 client.?.objecter ms_handle_connect 0x7ff85413f6a0 2024-04-04T21:02:30.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.062+0000 7ff842ffd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:30.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.065+0000 7f51daffd640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51dc0a9a50 0x7f51dc146540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:30.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.065+0000 7f51daffd640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51dc0a9a50 0x7f51dc146540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:33074/0 (socket says 172.21.15.106:33074) 2024-04-04T21:02:30.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.065+0000 7f51daffd640 1 -- 172.21.15.106:0/767903800 learned_addr learned my addr 172.21.15.106:0/767903800 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:30.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.065+0000 7f51daffd640 10 monclient(hunting): get_auth_request con 0x7f51dc0a9a50 auth_method 0 2024-04-04T21:02:30.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.065+0000 7f51daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:30.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.065+0000 7f51daffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:30.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.065+0000 7f51daffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:30.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.066+0000 7f51daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:30.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.066+0000 7f51daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:30.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.066+0000 7f51daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:30.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.069+0000 7f51daffd640 10 monclient(hunting): handle_auth_done global_id 4331 payload 824 2024-04-04T21:02:30.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.069+0000 7f51daffd640 1 -- 172.21.15.106:0/767903800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51dc13f6a0 msgr2=0x7f51dc1399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.069+0000 7f51daffd640 1 --2- 172.21.15.106:0/767903800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51dc13f6a0 0x7f51dc1399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.069+0000 7f51daffd640 1 -- 172.21.15.106:0/767903800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51dc142880 msgr2=0x7f51dc139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.069+0000 7f51daffd640 1 --2- 172.21.15.106:0/767903800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51dc142880 0x7f51dc139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.069+0000 7f51daffd640 10 monclient: _finish_hunting 0 2024-04-04T21:02:30.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.069+0000 7f51daffd640 1 monclient: found mon.noname-c 2024-04-04T21:02:30.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.069+0000 7f51daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:30.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.069+0000 7f51daffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:02:30.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.069+0000 7f51daffd640 1 -- 172.21.15.106:0/767903800 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc1dae30 con 0x7f51dc0a9a50 2024-04-04T21:02:30.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff842ffd640 1 -- 172.21.15.106:0/3094762490 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848017070 con 0x7ff85413f6a0 2024-04-04T21:02:30.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff842ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:30.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff842ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:30.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff842ffd640 10 monclient: dump: 2024-04-04T21:02:30.073 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:30.073 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:30.073 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.073 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.073 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:30.073 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:30.073 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:30.073 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:30.074 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:30.074 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:30.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff842ffd640 10 monclient: _finish_auth 0 2024-04-04T21:02:30.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff842ffd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:30.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff842ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:30.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff842ffd640 1 -- 172.21.15.106:0/3094762490 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8480035f0 con 0x7ff85413f6a0 2024-04-04T21:02:30.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff842ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:30.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff842ffd640 1 -- 172.21.15.106:0/3094762490 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848003a20 con 0x7ff85413f6a0 2024-04-04T21:02:30.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff842ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff842ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff842ffd640 10 monclient: dump: 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff858a78640 5 monclient: authenticate success, global_id 4328 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff858a78640 10 monclient: _renew_subs 2024-04-04T21:02:30.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff850d99640 4 set_mon_vals no callback set 2024-04-04T21:02:30.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff858a78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:30.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff858a78640 1 -- 172.21.15.106:0/3094762490 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8540ac7f0 con 0x7ff85413f6a0 2024-04-04T21:02:30.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff858a78640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:30.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff858a78640 10 monclient: _renew_subs 2024-04-04T21:02:30.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff858a78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:30.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff858a78640 1 -- 172.21.15.106:0/3094762490 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8540adad0 con 0x7ff85413f6a0 2024-04-04T21:02:30.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.072+0000 7ff858a78640 1 librados: init done 2024-04-04T21:02:30.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.073+0000 7f51daffd640 1 --2- 172.21.15.106:0/767903800 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51dc0a9a50 0x7f51dc146540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009990 tx=0x7f51cc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:30.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.073+0000 7f51c37fe640 10 client.?.objecter ms_handle_connect 0x7f51dc0a9a50 2024-04-04T21:02:30.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.073+0000 7f51c37fe640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:30.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.074+0000 7ff858a78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:30.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.074+0000 7ff858a78640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:30.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.074+0000 7ff858a78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:30.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.074+0000 7ff858a78640 1 -- 172.21.15.106:0/3094762490 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85404f050 con 0x7ff85413f6a0 2024-04-04T21:02:30.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.075+0000 7f19ebfe5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-04-04T21:02:30.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.075+0000 7f19ebfe5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-04-04T21:02:30.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.075+0000 7f19ebfe5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:30.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.075+0000 7f19ebfe5640 1 -- 172.21.15.106:0/1581971477 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f19e41cc780 con 0x7f19e40a9a50 2024-04-04T21:02:30.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.077+0000 7f51c37fe640 1 -- 172.21.15.106:0/767903800 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc032a20 con 0x7f51dc0a9a50 2024-04-04T21:02:30.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:30.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:30.080 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51c37fe640 10 monclient: dump: 2024-04-04T21:02:30.080 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:30.080 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:30.080 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.080 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.080 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:30.080 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:30.080 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:30.081 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:30.081 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:30.081 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:30.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51c37fe640 10 monclient: _finish_auth 0 2024-04-04T21:02:30.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51c37fe640 10 monclient: _check_auth_tickets 2024-04-04T21:02:30.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:30.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51e1671640 5 monclient: authenticate success, global_id 4331 2024-04-04T21:02:30.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51e1671640 10 monclient: _renew_subs 2024-04-04T21:02:30.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51e1671640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:30.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51e1671640 1 -- 172.21.15.106:0/767903800 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51dc13b6c0 con 0x7f51dc0a9a50 2024-04-04T21:02:30.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51e1671640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:30.081 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51e1671640 10 monclient: _renew_subs 2024-04-04T21:02:30.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51e1671640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:30.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51e1671640 1 -- 172.21.15.106:0/767903800 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51dc1db530 con 0x7f51dc0a9a50 2024-04-04T21:02:30.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.078+0000 7f51e1671640 1 librados: init done 2024-04-04T21:02:30.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.079+0000 7f51e1671640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:30.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.079+0000 7f51e1671640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:30.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.079+0000 7f51e1671640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:30.082 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.079+0000 7f51e1671640 1 -- 172.21.15.106:0/767903800 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51dc13b0c0 con 0x7f51dc0a9a50 2024-04-04T21:02:30.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.111+0000 7f4210ff9640 1 -- 172.21.15.106:0/1877687748 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f420c011070 con 0x7f42140b0fa0 2024-04-04T21:02:30.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.112+0000 7f4210ff9640 1 --2- 172.21.15.106:0/1877687748 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f420006ceb0 0x7f420006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:30.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.112+0000 7f4213fff640 1 --2- 172.21.15.106:0/1877687748 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f420006ceb0 0x7f420006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:30.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.112+0000 7f4213fff640 10 monclient: get_auth_request con 0x7f420006ceb0 auth_method 0 2024-04-04T21:02:30.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.113+0000 7f4213fff640 1 --2- 172.21.15.106:0/1877687748 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f420006ceb0 0x7f420006f370 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4204033000 tx=0x7f4204005000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:30.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.113+0000 7f4210ff9640 10 client.4350.objecter ms_handle_connect 0x7f420006ceb0 2024-04-04T21:02:30.130 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.128+0000 7f4210ff9640 1 -- 172.21.15.106:0/1877687748 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f420c0565a0 con 0x7f42140b0fa0 2024-04-04T21:02:30.130 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.128+0000 7f4210ff9640 10 client.4350.objecter ms_dispatch 0x7f4214000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:30.130 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.128+0000 7f4210ff9640 3 client.4350.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:30.130 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.128+0000 7f4210ff9640 3 client.4350.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:30.130 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.129+0000 7f4210ff9640 20 client.4350.objecter dump_active .. 0 homeless 2024-04-04T21:02:30.130 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.129+0000 7f4210ff9640 1 -- 172.21.15.106:0/1877687748 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f420c01a060 con 0x7f42140b0fa0 2024-04-04T21:02:30.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.129+0000 7f4210ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:30.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.129+0000 7f4210ff9640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:30.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.150+0000 7f3d837fe640 1 -- 172.21.15.106:0/2655651918 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3d8c058170 con 0x7f3d9c0a9a50 2024-04-04T21:02:30.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.150+0000 7f3d837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:30.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.150+0000 7f3d837fe640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:30.194 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.194+0000 7f5fecca5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-04T21:02:30.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.194+0000 7f5fecca5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-04T21:02:30.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.194+0000 7f5fecca5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:30.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.194+0000 7f5fecca5640 1 -- 172.21.15.106:0/3900206976 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f5fe804f050 con 0x7f5fe81d7fa0 2024-04-04T21:02:30.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.198+0000 7f5fd6ffd640 1 -- 172.21.15.106:0/3900206976 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5fe007b430 con 0x7f5fe81d7fa0 2024-04-04T21:02:30.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.198+0000 7f5fd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-04T21:02:30.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.198+0000 7f5fd6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:30.199 INFO:teuthology.orchestra.run.smithi106.stdout:55834574853 2024-04-04T21:02:30.201 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.200+0000 7f5fecca5640 10 librados: watch_flush enter 2024-04-04T21:02:30.201 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.200+0000 7f5fecca5640 10 librados: watch_flush exit 2024-04-04T21:02:30.201 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.201+0000 7f5fecca5640 20 client.4390.objecter shutdown clearing up homeless session... 2024-04-04T21:02:30.201 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.201+0000 7f5fecca5640 10 client.4390.objecter successfully canceled tick 2024-04-04T21:02:30.201 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.201+0000 7f5fecca5640 1 -- 172.21.15.106:0/3900206976 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5fc806ceb0 msgr2=0x7f5fc806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.201 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.201+0000 7f5fecca5640 1 --2- 172.21.15.106:0/3900206976 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5fc806ceb0 0x7f5fc806f370 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8001eb0 tx=0x7f5fd8001600 comp rx=0 tx=0).stop 2024-04-04T21:02:30.202 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.201+0000 7f5fecca5640 10 monclient: shutdown 2024-04-04T21:02:30.202 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.201+0000 7f5fecca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:30.202 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.201+0000 7f5fecca5640 1 -- 172.21.15.106:0/3900206976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe81d7fa0 msgr2=0x7f5fe81daac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.202 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.201+0000 7f5fecca5640 1 --2- 172.21.15.106:0/3900206976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe81d7fa0 0x7f5fe81daac0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0032040 tx=0x7f5fe0001330 comp rx=0 tx=0).stop 2024-04-04T21:02:30.202 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.201+0000 7f5fecca5640 1 -- 172.21.15.106:0/3900206976 shutdown_connections 2024-04-04T21:02:30.202 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.201+0000 7f5fecca5640 1 --2- 172.21.15.106:0/3900206976 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5fc806ceb0 0x7f5fc806f370 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.202 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.201+0000 7f5fecca5640 1 --2- 172.21.15.106:0/3900206976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fe81d7fa0 0x7f5fe81daac0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.202 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.201+0000 7f5fecca5640 1 --2- 172.21.15.106:0/3900206976 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5fe8146620 0x7f5fe81d7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.202 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.201+0000 7f5fecca5640 1 --2- 172.21.15.106:0/3900206976 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fd0004e60 0x7f5fe80ac080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.202 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.201+0000 7f5fecca5640 1 -- 172.21.15.106:0/3900206976 >> 172.21.15.106:0/3900206976 conn(0x7f5fe80b8030 msgr2=0x7f5fe80b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:30.203 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.202+0000 7f5fecca5640 1 -- 172.21.15.106:0/3900206976 shutdown_connections 2024-04-04T21:02:30.203 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.202+0000 7f5fecca5640 1 -- 172.21.15.106:0/3900206976 wait complete. 2024-04-04T21:02:30.203 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.202+0000 7f5fecca5640 1 librados: shutdown 2024-04-04T21:02:30.218 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-04-04T21:02:30.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.269+0000 7fe3f1667640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-04T21:02:30.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.269+0000 7fe3f1667640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-04T21:02:30.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.269+0000 7fe3f1667640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:30.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.269+0000 7fe3f1667640 1 -- 172.21.15.106:0/3731565926 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe3ec067ed0 con 0x7fe3ec13f6a0 2024-04-04T21:02:30.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.334+0000 7f19da7fc640 1 -- 172.21.15.106:0/1581971477 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f19e007b650 con 0x7f19e40a9a50 2024-04-04T21:02:30.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.334+0000 7f19da7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-04-04T21:02:30.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.334+0000 7f19da7fc640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:30.335 INFO:teuthology.orchestra.run.smithi106.stdout:64424509447 2024-04-04T21:02:30.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.337+0000 7f19ebfe5640 10 librados: watch_flush enter 2024-04-04T21:02:30.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.337+0000 7f19ebfe5640 10 librados: watch_flush exit 2024-04-04T21:02:30.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.337+0000 7f19ebfe5640 20 client.4344.objecter shutdown clearing up homeless session... 2024-04-04T21:02:30.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.337+0000 7f19ebfe5640 10 client.4344.objecter successfully canceled tick 2024-04-04T21:02:30.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.337+0000 7f19ebfe5640 1 -- 172.21.15.106:0/1581971477 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f19c806ceb0 msgr2=0x7f19c806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.337+0000 7f19ebfe5640 1 --2- 172.21.15.106:0/1581971477 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f19c806ceb0 0x7f19c806f370 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f19d4000f60 tx=0x7f19d40079d0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.338+0000 7f19ebfe5640 10 monclient: shutdown 2024-04-04T21:02:30.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.338+0000 7f19ebfe5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:30.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.338+0000 7f19ebfe5640 1 -- 172.21.15.106:0/1581971477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19e40a9a50 msgr2=0x7f19e41cf200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.338+0000 7f19ebfe5640 1 --2- 172.21.15.106:0/1581971477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19e40a9a50 0x7f19e41cf200 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f19e002bee0 tx=0x7f19e00046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.338+0000 7f19ebfe5640 1 -- 172.21.15.106:0/1581971477 shutdown_connections 2024-04-04T21:02:30.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.338+0000 7f19ebfe5640 1 --2- 172.21.15.106:0/1581971477 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f19c806ceb0 0x7f19c806f370 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.338+0000 7f19ebfe5640 1 --2- 172.21.15.106:0/1581971477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19e413fac0 0x7f19e41ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.338+0000 7f19ebfe5640 1 --2- 172.21.15.106:0/1581971477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19e413f6a0 0x7f19e41cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.338+0000 7f19ebfe5640 1 --2- 172.21.15.106:0/1581971477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19e40a9a50 0x7f19e41cf200 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.338+0000 7f19ebfe5640 1 -- 172.21.15.106:0/1581971477 >> 172.21.15.106:0/1581971477 conn(0x7f19e40b8030 msgr2=0x7f19e40b5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:30.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.338+0000 7f19ebfe5640 1 -- 172.21.15.106:0/1581971477 shutdown_connections 2024-04-04T21:02:30.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.338+0000 7f19ebfe5640 1 -- 172.21.15.106:0/1581971477 wait complete. 2024-04-04T21:02:30.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.338+0000 7f19ebfe5640 1 librados: shutdown 2024-04-04T21:02:30.350 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509447 for osd.6 2024-04-04T21:02:30.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.354+0000 7f3da1182640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-04T21:02:30.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.354+0000 7f3da1182640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-04T21:02:30.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.354+0000 7f3da1182640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:30.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.354+0000 7f3da1182640 1 -- 172.21.15.106:0/2655651918 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f3d9c1d7950 con 0x7f3d9c0a9a50 2024-04-04T21:02:30.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.358+0000 7f3d837fe640 1 -- 172.21.15.106:0/2655651918 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3d8c05c210 con 0x7f3d9c0a9a50 2024-04-04T21:02:30.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.358+0000 7f3d837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-04T21:02:30.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.358+0000 7f3d837fe640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:30.359 INFO:teuthology.orchestra.run.smithi106.stdout:60129542150 2024-04-04T21:02:30.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 10 librados: watch_flush enter 2024-04-04T21:02:30.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 10 librados: watch_flush exit 2024-04-04T21:02:30.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 20 client.4396.objecter shutdown clearing up homeless session... 2024-04-04T21:02:30.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 10 client.4396.objecter successfully canceled tick 2024-04-04T21:02:30.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 1 -- 172.21.15.106:0/2655651918 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f3d8806cee0 msgr2=0x7f3d8806f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 1 --2- 172.21.15.106:0/2655651918 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f3d8806cee0 0x7f3d8806f3a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c13f2c0 tx=0x7f3d90006c90 comp rx=0 tx=0).stop 2024-04-04T21:02:30.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 10 monclient: shutdown 2024-04-04T21:02:30.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:30.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 1 -- 172.21.15.106:0/2655651918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c0a9a50 msgr2=0x7f3d9c146570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 1 --2- 172.21.15.106:0/2655651918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c0a9a50 0x7f3d9c146570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c00c4d0 tx=0x7f3d8c00c990 comp rx=0 tx=0).stop 2024-04-04T21:02:30.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 1 -- 172.21.15.106:0/2655651918 shutdown_connections 2024-04-04T21:02:30.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 1 --2- 172.21.15.106:0/2655651918 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f3d8806cee0 0x7f3d8806f3a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 1 --2- 172.21.15.106:0/2655651918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d9c142dc0 0x7f3d9c1d7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 1 --2- 172.21.15.106:0/2655651918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d9c0a9e30 0x7f3d9c1445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 1 --2- 172.21.15.106:0/2655651918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c0a9a50 0x7f3d9c146570 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 1 -- 172.21.15.106:0/2655651918 >> 172.21.15.106:0/2655651918 conn(0x7f3d9c0b8030 msgr2=0x7f3d9c0b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:30.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 1 -- 172.21.15.106:0/2655651918 shutdown_connections 2024-04-04T21:02:30.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 1 -- 172.21.15.106:0/2655651918 wait complete. 2024-04-04T21:02:30.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.361+0000 7f3da1182640 1 librados: shutdown 2024-04-04T21:02:30.372 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.7 2024-04-04T21:02:30.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.402+0000 7f421a506640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-04T21:02:30.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.402+0000 7f421a506640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-04T21:02:30.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.402+0000 7f421a506640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:30.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.402+0000 7f421a506640 1 -- 172.21.15.106:0/1877687748 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f421413df80 con 0x7f42140b0fa0 2024-04-04T21:02:30.404 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.404+0000 7ff842ffd640 1 -- 172.21.15.106:0/3094762490 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff848003bc0 con 0x7ff85413f6a0 2024-04-04T21:02:30.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.405+0000 7ff842ffd640 1 --2- 172.21.15.106:0/3094762490 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff83406ce60 0x7ff83406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:30.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.405+0000 7ff85259c640 1 --2- 172.21.15.106:0/3094762490 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff83406ce60 0x7ff83406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:30.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.405+0000 7ff85259c640 10 monclient: get_auth_request con 0x7ff83406ce60 auth_method 0 2024-04-04T21:02:30.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.405+0000 7ff842ffd640 10 client.4328.objecter ms_handle_connect 0x7ff83406ce60 2024-04-04T21:02:30.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.405+0000 7ff85259c640 1 --2- 172.21.15.106:0/3094762490 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff83406ce60 0x7ff83406f320 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff844036000 tx=0x7ff844030000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:30.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.407+0000 7f4210ff9640 1 -- 172.21.15.106:0/1877687748 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f420c00f260 con 0x7f42140b0fa0 2024-04-04T21:02:30.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.407+0000 7f4210ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-04T21:02:30.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.407+0000 7f4210ff9640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:30.408 INFO:teuthology.orchestra.run.smithi106.stdout:85899345928 2024-04-04T21:02:30.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 10 librados: watch_flush enter 2024-04-04T21:02:30.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 10 librados: watch_flush exit 2024-04-04T21:02:30.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 20 client.4350.objecter shutdown clearing up homeless session... 2024-04-04T21:02:30.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 10 client.4350.objecter successfully canceled tick 2024-04-04T21:02:30.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 1 -- 172.21.15.106:0/1877687748 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f420006ceb0 msgr2=0x7f420006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 1 --2- 172.21.15.106:0/1877687748 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f420006ceb0 0x7f420006f370 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4204033000 tx=0x7f4204005000 comp rx=0 tx=0).stop 2024-04-04T21:02:30.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 10 monclient: shutdown 2024-04-04T21:02:30.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:30.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 1 -- 172.21.15.106:0/1877687748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42140b0fa0 msgr2=0x7f42141d7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 1 --2- 172.21.15.106:0/1877687748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42140b0fa0 0x7f42141d7c60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f42140b48a0 tx=0x7f420c00ccb0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 1 -- 172.21.15.106:0/1877687748 shutdown_connections 2024-04-04T21:02:30.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 1 --2- 172.21.15.106:0/1877687748 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f420006ceb0 0x7f420006f370 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 1 --2- 172.21.15.106:0/1877687748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42140b0fa0 0x7f42141d7c60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 1 --2- 172.21.15.106:0/1877687748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42141db910 0x7f42140b0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 1 --2- 172.21.15.106:0/1877687748 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc003a00 0x7f42141db3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 1 -- 172.21.15.106:0/1877687748 >> 172.21.15.106:0/1877687748 conn(0x7f42140b7ec0 msgr2=0x7f421413ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:30.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.409+0000 7f421a506640 1 -- 172.21.15.106:0/1877687748 shutdown_connections 2024-04-04T21:02:30.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.410+0000 7f421a506640 1 -- 172.21.15.106:0/1877687748 wait complete. 2024-04-04T21:02:30.412 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.410+0000 7f421a506640 1 librados: shutdown 2024-04-04T21:02:30.421 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345928 for osd.1 2024-04-04T21:02:30.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.645+0000 7ff842ffd640 1 -- 172.21.15.106:0/3094762490 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff8480562c0 con 0x7ff85413f6a0 2024-04-04T21:02:30.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.645+0000 7ff842ffd640 10 client.4328.objecter ms_dispatch 0x7ff854000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:30.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.645+0000 7ff842ffd640 3 client.4328.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:30.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.645+0000 7ff842ffd640 3 client.4328.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:30.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.645+0000 7ff842ffd640 20 client.4328.objecter dump_active .. 0 homeless 2024-04-04T21:02:30.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.645+0000 7ff842ffd640 1 -- 172.21.15.106:0/3094762490 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff848020020 con 0x7ff85413f6a0 2024-04-04T21:02:30.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.646+0000 7ff842ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:30.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.646+0000 7ff842ffd640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:30.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.662+0000 7f51c37fe640 1 -- 172.21.15.106:0/767903800 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc037020 con 0x7f51dc0a9a50 2024-04-04T21:02:30.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.662+0000 7f51c37fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:30.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.662+0000 7f51c37fe640 1 -- 172.21.15.106:0/767903800 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc032de0 con 0x7f51dc0a9a50 2024-04-04T21:02:30.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.662+0000 7f51c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:30.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.662+0000 7f51c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:30.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.662+0000 7f51c37fe640 10 monclient: dump: 2024-04-04T21:02:30.663 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:30.663 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:30.663 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.663 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:30.663 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:30.663 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:30.663 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:30.663 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:30.664 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:30.664 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:30.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.662+0000 7f51d97fa640 4 set_mon_vals no callback set 2024-04-04T21:02:30.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.663+0000 7fe3d37fe640 1 -- 172.21.15.106:0/3731565926 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe3e007acf0 con 0x7fe3ec13f6a0 2024-04-04T21:02:30.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.663+0000 7fe3d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-04T21:02:30.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.663+0000 7fe3d37fe640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:30.665 INFO:teuthology.orchestra.run.smithi106.stdout:85899345927 2024-04-04T21:02:30.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.663+0000 7f51c37fe640 1 -- 172.21.15.106:0/767903800 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f51cc04c060 con 0x7f51dc0a9a50 2024-04-04T21:02:30.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.663+0000 7f51c37fe640 1 --2- 172.21.15.106:0/767903800 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f51c806ce60 0x7f51c806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:30.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.664+0000 7f51da7fc640 1 --2- 172.21.15.106:0/767903800 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f51c806ce60 0x7f51c806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:30.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.664+0000 7f51da7fc640 10 monclient: get_auth_request con 0x7f51c806ce60 auth_method 0 2024-04-04T21:02:30.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.664+0000 7f51c37fe640 10 client.4331.objecter ms_handle_connect 0x7f51c806ce60 2024-04-04T21:02:30.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.664+0000 7f51da7fc640 1 --2- 172.21.15.106:0/767903800 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f51c806ce60 0x7f51c806f320 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f51dc13b450 tx=0x7f51d0006c90 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:30.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.665+0000 7fe3f1667640 10 librados: watch_flush enter 2024-04-04T21:02:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.665+0000 7fe3f1667640 10 librados: watch_flush exit 2024-04-04T21:02:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.665+0000 7fe3f1667640 20 client.4319.objecter shutdown clearing up homeless session... 2024-04-04T21:02:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.665+0000 7fe3f1667640 10 client.4319.objecter successfully canceled tick 2024-04-04T21:02:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 1 -- 172.21.15.106:0/3731565926 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe3d806ceb0 msgr2=0x7fe3d806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 1 --2- 172.21.15.106:0/3731565926 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe3d806ceb0 0x7fe3d806f370 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc009710 tx=0x7fe3dc006c90 comp rx=0 tx=0).stop 2024-04-04T21:02:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 10 monclient: shutdown 2024-04-04T21:02:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 1 -- 172.21.15.106:0/3731565926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3ec13f6a0 msgr2=0x7fe3ec1460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 1 --2- 172.21.15.106:0/3731565926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3ec13f6a0 0x7fe3ec1460c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0000c00 tx=0x7fe3e00046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 1 -- 172.21.15.106:0/3731565926 shutdown_connections 2024-04-04T21:02:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 1 --2- 172.21.15.106:0/3731565926 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe3d806ceb0 0x7fe3d806f370 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 1 --2- 172.21.15.106:0/3731565926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3ec142880 0x7fe3ec146600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 1 --2- 172.21.15.106:0/3731565926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3ec13f6a0 0x7fe3ec1460c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 1 --2- 172.21.15.106:0/3731565926 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3ec0a9a50 0x7fe3ec135940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 1 -- 172.21.15.106:0/3731565926 >> 172.21.15.106:0/3731565926 conn(0x7fe3ec0b8030 msgr2=0x7fe3ec142cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:30.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 1 -- 172.21.15.106:0/3731565926 shutdown_connections 2024-04-04T21:02:30.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 1 -- 172.21.15.106:0/3731565926 wait complete. 2024-04-04T21:02:30.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.666+0000 7fe3f1667640 1 librados: shutdown 2024-04-04T21:02:30.678 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345928 got 85899345927 for osd.3 2024-04-04T21:02:30.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.679+0000 7f51c37fe640 1 -- 172.21.15.106:0/767903800 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f51cc0786e0 con 0x7f51dc0a9a50 2024-04-04T21:02:30.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.679+0000 7f51c37fe640 10 client.4331.objecter ms_dispatch 0x7f51dc000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:30.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.679+0000 7f51c37fe640 3 client.4331.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:30.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.679+0000 7f51c37fe640 3 client.4331.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:30.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.679+0000 7f51c37fe640 20 client.4331.objecter dump_active .. 0 homeless 2024-04-04T21:02:30.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.679+0000 7f51c37fe640 1 -- 172.21.15.106:0/767903800 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f51dc13b0c0 con 0x7f51dc0a9a50 2024-04-04T21:02:30.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.679+0000 7f51c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:30.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.679+0000 7f51c37fe640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:30.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.839+0000 7ff858a78640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-04T21:02:30.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.839+0000 7ff858a78640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-04T21:02:30.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.839+0000 7ff858a78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:30.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.839+0000 7ff858a78640 1 -- 172.21.15.106:0/3094762490 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff8540b47f0 con 0x7ff85413f6a0 2024-04-04T21:02:30.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.844+0000 7ff842ffd640 1 -- 172.21.15.106:0/3094762490 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff848058920 con 0x7ff85413f6a0 2024-04-04T21:02:30.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.844+0000 7ff842ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-04T21:02:30.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.844+0000 7ff842ffd640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:30.845 INFO:teuthology.orchestra.run.smithi106.stdout:85899345928 2024-04-04T21:02:30.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 10 librados: watch_flush enter 2024-04-04T21:02:30.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 10 librados: watch_flush exit 2024-04-04T21:02:30.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 20 client.4328.objecter shutdown clearing up homeless session... 2024-04-04T21:02:30.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 10 client.4328.objecter successfully canceled tick 2024-04-04T21:02:30.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 1 -- 172.21.15.106:0/3094762490 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff83406ce60 msgr2=0x7ff83406f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 1 --2- 172.21.15.106:0/3094762490 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff83406ce60 0x7ff83406f320 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff844036000 tx=0x7ff844030000 comp rx=0 tx=0).stop 2024-04-04T21:02:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 10 monclient: shutdown 2024-04-04T21:02:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 1 -- 172.21.15.106:0/3094762490 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff85413f6a0 msgr2=0x7ff8541d87c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 1 --2- 172.21.15.106:0/3094762490 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff85413f6a0 0x7ff8541d87c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff84800c410 tx=0x7ff84800c8d0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 1 -- 172.21.15.106:0/3094762490 shutdown_connections 2024-04-04T21:02:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 1 --2- 172.21.15.106:0/3094762490 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff83406ce60 0x7ff83406f320 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 1 --2- 172.21.15.106:0/3094762490 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff85413fac0 0x7ff8540ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 1 --2- 172.21.15.106:0/3094762490 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff85413f6a0 0x7ff8541d87c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 1 --2- 172.21.15.106:0/3094762490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8540a9a50 0x7ff8541d8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 1 -- 172.21.15.106:0/3094762490 >> 172.21.15.106:0/3094762490 conn(0x7ff8540b8030 msgr2=0x7ff854142880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 1 -- 172.21.15.106:0/3094762490 shutdown_connections 2024-04-04T21:02:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 1 -- 172.21.15.106:0/3094762490 wait complete. 2024-04-04T21:02:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.846+0000 7ff858a78640 1 librados: shutdown 2024-04-04T21:02:30.858 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345928 for osd.2 2024-04-04T21:02:30.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.888+0000 7f51e1671640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-04T21:02:30.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.888+0000 7f51e1671640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-04T21:02:30.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.888+0000 7f51e1671640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:30.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.888+0000 7f51e1671640 1 -- 172.21.15.106:0/767903800 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f51dc1db000 con 0x7f51dc0a9a50 2024-04-04T21:02:30.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.907+0000 7f51c37fe640 1 -- 172.21.15.106:0/767903800 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f51cc040070 con 0x7f51dc0a9a50 2024-04-04T21:02:30.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.907+0000 7f51c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-04T21:02:30.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.907+0000 7f51c37fe640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:30.908 INFO:teuthology.orchestra.run.smithi106.stdout:77309411338 2024-04-04T21:02:30.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 10 librados: watch_flush enter 2024-04-04T21:02:30.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 10 librados: watch_flush exit 2024-04-04T21:02:30.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 20 client.4331.objecter shutdown clearing up homeless session... 2024-04-04T21:02:30.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 10 client.4331.objecter successfully canceled tick 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 1 -- 172.21.15.106:0/767903800 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f51c806ce60 msgr2=0x7f51c806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 1 --2- 172.21.15.106:0/767903800 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f51c806ce60 0x7f51c806f320 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f51dc13b450 tx=0x7f51d0006c90 comp rx=0 tx=0).stop 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 10 monclient: shutdown 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 1 -- 172.21.15.106:0/767903800 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51dc0a9a50 msgr2=0x7f51dc146540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 1 --2- 172.21.15.106:0/767903800 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51dc0a9a50 0x7f51dc146540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009990 tx=0x7f51cc0046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 1 -- 172.21.15.106:0/767903800 shutdown_connections 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 1 --2- 172.21.15.106:0/767903800 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f51c806ce60 0x7f51c806f320 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 1 --2- 172.21.15.106:0/767903800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51dc142880 0x7f51dc139ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 1 --2- 172.21.15.106:0/767903800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51dc13f6a0 0x7f51dc1399b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 1 --2- 172.21.15.106:0/767903800 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51dc0a9a50 0x7f51dc146540 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 1 -- 172.21.15.106:0/767903800 >> 172.21.15.106:0/767903800 conn(0x7f51dc0b8030 msgr2=0x7f51dc0b5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 1 -- 172.21.15.106:0/767903800 shutdown_connections 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 1 -- 172.21.15.106:0/767903800 wait complete. 2024-04-04T21:02:30.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:30.909+0000 7f51e1671640 1 librados: shutdown 2024-04-04T21:02:30.921 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411338 got 77309411338 for osd.4 2024-04-04T21:02:30.921 DEBUG:teuthology.parallel:result is None 2024-04-04T21:02:31.041 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-04T21:02:31.141 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.141+0000 7f657e36a640 10 monclient: get_monmap_and_config 2024-04-04T21:02:31.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.141+0000 7f657e36a640 10 monclient: build_initial_monmap 2024-04-04T21:02:31.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.141+0000 7f657e36a640 10 monclient: monmap: 2024-04-04T21:02:31.142 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:31.142 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.142 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:31.142383+0000 2024-04-04T21:02:31.142 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:31.142383+0000 2024-04-04T21:02:31.142 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:31.142 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.142 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:31.143 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:31.143 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:31.143 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:31.143 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:31.143 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 1 Processor -- start 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 1 -- start start 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 10 monclient: init 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65780b0b10 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65780b14b0 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65780ae460 0x7f6578072b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 10 monclient(hunting): picked mon.noname-b con 0x7f65780ae460 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65781464f0 con 0x7f65780b0b10 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65780ad9c0 con 0x7f65780b14b0 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65780b0ef0 con 0x7f65780ae460 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.142+0000 7f657e36a640 10 monclient(hunting): authenticate will time out at 1400.409875s 2024-04-04T21:02:31.145 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.143+0000 7f6577fff640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f65780b0b10 0x7f6578145de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:49596/0 (socket says 172.21.15.106:49596) 2024-04-04T21:02:31.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.143+0000 7f6577fff640 1 -- 172.21.15.106:0/3785330028 learned_addr learned my addr 172.21.15.106:0/3785330028 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:31.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.143+0000 7f657c907640 1 --2- 172.21.15.106:0/3785330028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65780ae460 0x7f6578072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.143+0000 7f657c907640 10 monclient(hunting): get_auth_request con 0x7f65780ae460 auth_method 0 2024-04-04T21:02:31.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.143+0000 7f657c907640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.143+0000 7f657c907640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.143+0000 7f657c907640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.144+0000 7f657c907640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.144+0000 7f657c907640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.146 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.144+0000 7f657c907640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.145+0000 7f6576ffd640 1 -- 172.21.15.106:0/3785330028 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440112526 0 0) 0x7f65781464f0 con 0x7f65780b0b10 2024-04-04T21:02:31.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.145+0000 7f6576ffd640 10 monclient(hunting): _negotiate 2024-04-04T21:02:31.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.145+0000 7f6576ffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.145+0000 7f6576ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.145+0000 7f6576ffd640 10 monclient(hunting): my global_id is 4353 2024-04-04T21:02:31.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.145+0000 7f6576ffd640 1 -- 172.21.15.106:0/3785330028 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6560003520 con 0x7f65780b0b10 2024-04-04T21:02:31.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.145+0000 7f657c907640 10 monclient(hunting): handle_auth_done global_id 4334 payload 274 2024-04-04T21:02:31.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.145+0000 7f657c907640 1 -- 172.21.15.106:0/3785330028 >> v1:172.21.15.203:6789/0 conn(0x7f65780b14b0 legacy=0x7f657813fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.145+0000 7f657c907640 1 -- 172.21.15.106:0/3785330028 >> v1:172.21.15.106:6789/0 conn(0x7f65780b0b10 legacy=0x7f6578145de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.145+0000 7f657c907640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.145+0000 7f657c907640 1 monclient: found mon.noname-b 2024-04-04T21:02:31.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.145+0000 7f657c907640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.145+0000 7f657c907640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.145+0000 7f657c907640 1 -- 172.21.15.106:0/3785330028 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65781403b0 con 0x7f65780ae460 2024-04-04T21:02:31.148 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.147+0000 7f657c907640 1 --2- 172.21.15.106:0/3785330028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65780ae460 0x7f6578072b60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6570009dc0 tx=0x7f657002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c62d74338b9d4d28 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f6576ffd640 1 -- 172.21.15.106:0/3785330028 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570039070 con 0x7f65780ae460 2024-04-04T21:02:31.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f6576ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f6576ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:31.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f6576ffd640 10 monclient: dump: 2024-04-04T21:02:31.152 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.152 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.152 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.152 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.152 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.152 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.152 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f6576ffd640 10 monclient: _finish_auth 0 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f6576ffd640 10 monclient: _check_auth_tickets 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f6576ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f6576ffd640 1 -- 172.21.15.106:0/3785330028 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657002cc40 con 0x7f65780ae460 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f657e36a640 5 monclient: authenticate success, global_id 4334 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f657e36a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f6576ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f657e36a640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f6576ffd640 1 -- 172.21.15.106:0/3785330028 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570034440 con 0x7f65780ae460 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f657e36a640 4 set_mon_vals no callback set 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f6576ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f6576ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:31.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f6576ffd640 10 monclient: dump: 2024-04-04T21:02:31.154 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.154 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.154 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.154 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.154 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.154 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.154 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.154 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.154 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.154 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f657e36a640 10 monclient: shutdown 2024-04-04T21:02:31.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f657e36a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:31.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f657e36a640 1 -- 172.21.15.106:0/3785330028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65780ae460 msgr2=0x7f6578072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f657e36a640 1 --2- 172.21.15.106:0/3785330028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65780ae460 0x7f6578072b60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6570009dc0 tx=0x7f657002be90 comp rx=0 tx=0).stop 2024-04-04T21:02:31.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f657e36a640 1 -- 172.21.15.106:0/3785330028 shutdown_connections 2024-04-04T21:02:31.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f657e36a640 1 --2- 172.21.15.106:0/3785330028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65780ae460 0x7f6578072b60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f657e36a640 1 -- 172.21.15.106:0/3785330028 >> 172.21.15.106:0/3785330028 conn(0x7f65780b1990 msgr2=0x7f65780b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:31.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f657e36a640 1 -- 172.21.15.106:0/3785330028 shutdown_connections 2024-04-04T21:02:31.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.151+0000 7f657e36a640 1 -- 172.21.15.106:0/3785330028 wait complete. 2024-04-04T21:02:31.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.152+0000 7f657e36a640 10 monclient: build_initial_monmap 2024-04-04T21:02:31.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.152+0000 7f657e36a640 10 monclient: monmap: 2024-04-04T21:02:31.158 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:31.158 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:31.158 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:31.158 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:31.158 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:31.158 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.152+0000 7f657e36a640 1 librados: starting msgr at 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.152+0000 7f657e36a640 1 librados: starting objecter 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.152+0000 7f657e36a640 1 Processor -- start 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 1 -- start start 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 1 librados: setting wanted keys 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 1 librados: calling monclient init 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 10 monclient: init 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65780ae460 0x7f6578145650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.159 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 10 monclient(hunting): picked mon.noname-a con 0x7f65780ae460 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65780b0b10 0x7f6578142b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 10 monclient(hunting): picked mon.noname-c con 0x7f65780b0b10 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65780b14b0 0x7f6578136f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 10 monclient(hunting): picked mon.noname-b con 0x7f65780b14b0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657814e330 con 0x7f65780ae460 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65780add30 con 0x7f65780b14b0 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657814e4b0 con 0x7f65780b0b10 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.153+0000 7f657e36a640 10 monclient(hunting): authenticate will time out at 1400.420876s 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f6577fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65780ae460 0x7f6578145650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f65777fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65780b0b10 0x7f6578142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f6577fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65780ae460 0x7f6578145650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35110/0 (socket says 172.21.15.106:35110) 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f6577fff640 1 -- 172.21.15.106:0/2383310870 learned_addr learned my addr 172.21.15.106:0/2383310870 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f6577fff640 10 monclient(hunting): get_auth_request con 0x7f65780ae460 auth_method 0 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f6577fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f6577fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f6577fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f65777fe640 10 monclient(hunting): get_auth_request con 0x7f65780b0b10 auth_method 0 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f65777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f65777fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f65777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f657c907640 1 --2- 172.21.15.106:0/2383310870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65780b14b0 0x7f6578136f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f657c907640 10 monclient(hunting): get_auth_request con 0x7f65780b14b0 auth_method 0 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f657c907640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f657c907640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.154+0000 7f657c907640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.161 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.155+0000 7f6577fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.155+0000 7f6577fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.155+0000 7f6577fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.155+0000 7f65777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.155+0000 7f65777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.155+0000 7f65777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.155+0000 7f657c907640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.155+0000 7f657c907640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.155+0000 7f657c907640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.157+0000 7f65777fe640 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2024-04-04T21:02:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.157+0000 7f65777fe640 1 -- 172.21.15.106:0/2383310870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65780b14b0 msgr2=0x7f6578136f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.157+0000 7f65777fe640 1 --2- 172.21.15.106:0/2383310870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65780b14b0 0x7f6578136f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.157+0000 7f65777fe640 1 -- 172.21.15.106:0/2383310870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65780ae460 msgr2=0x7f6578145650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.157+0000 7f65777fe640 1 --2- 172.21.15.106:0/2383310870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65780ae460 0x7f6578145650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.157+0000 7f65777fe640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.157+0000 7f65777fe640 1 monclient: found mon.noname-c 2024-04-04T21:02:31.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.157+0000 7f65777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.157+0000 7f65777fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.157+0000 7f65777fe640 1 -- 172.21.15.106:0/2383310870 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657813c5f0 con 0x7f65780b0b10 2024-04-04T21:02:31.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.159+0000 7f65777fe640 1 --2- 172.21.15.106:0/2383310870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65780b0b10 0x7f6578142b80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f656800c3f0 tx=0x7f656800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.159+0000 7f6574ff9640 10 client.?.objecter ms_handle_connect 0x7f65780b0b10 2024-04-04T21:02:31.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.159+0000 7f6574ff9640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:31.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.161+0000 7f6574ff9640 1 -- 172.21.15.106:0/2383310870 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65680052d0 con 0x7f65780b0b10 2024-04-04T21:02:31.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.161+0000 7f6574ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.161+0000 7f6574ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:31.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.161+0000 7f6574ff9640 10 monclient: dump: 2024-04-04T21:02:31.165 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.165 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.165 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.165 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.165 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.165 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.165 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.165 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.161+0000 7f6574ff9640 10 monclient: _finish_auth 0 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.161+0000 7f6574ff9640 10 monclient: _check_auth_tickets 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.161+0000 7f6574ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.161+0000 7f657e36a640 5 monclient: authenticate success, global_id 4337 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.161+0000 7f657e36a640 10 monclient: _renew_subs 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.161+0000 7f657e36a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.161+0000 7f657e36a640 1 -- 172.21.15.106:0/2383310870 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65781430c0 con 0x7f65780b0b10 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.162+0000 7f657e36a640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.162+0000 7f657e36a640 10 monclient: _renew_subs 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.162+0000 7f657e36a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.162+0000 7f657e36a640 1 -- 172.21.15.106:0/2383310870 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65781374a0 con 0x7f65780b0b10 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.162+0000 7f657e36a640 1 librados: init done 2024-04-04T21:02:31.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.162+0000 7f657e36a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.162+0000 7f657e36a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.162+0000 7f657e36a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.162+0000 7f657e36a640 1 -- 172.21.15.106:0/2383310870 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657804f050 con 0x7f65780b0b10 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.163+0000 7f6574ff9640 1 -- 172.21.15.106:0/2383310870 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6568005470 con 0x7f65780b0b10 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.163+0000 7f6574ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.163+0000 7f6574ff9640 1 -- 172.21.15.106:0/2383310870 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568020060 con 0x7f65780b0b10 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.163+0000 7f6574ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.163+0000 7f6574ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.163+0000 7f6574ff9640 10 monclient: dump: 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.167 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.168 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.168 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.168 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.168 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.168 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.163+0000 7f65767fc640 4 set_mon_vals no callback set 2024-04-04T21:02:31.218 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-04T21:02:31.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.319+0000 7f659fdfb640 10 monclient: get_monmap_and_config 2024-04-04T21:02:31.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.319+0000 7f659fdfb640 10 monclient: build_initial_monmap 2024-04-04T21:02:31.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.319+0000 7f659fdfb640 10 monclient: monmap: 2024-04-04T21:02:31.320 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:31.320 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.320 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:31.320762+0000 2024-04-04T21:02:31.320 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:31.320762+0000 2024-04-04T21:02:31.320 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:31.320 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.320 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:31.321 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:31.321 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:31.321 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:31.321 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:31.321 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.320+0000 7f659fdfb640 1 Processor -- start 2024-04-04T21:02:31.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.320+0000 7f659fdfb640 1 -- start start 2024-04-04T21:02:31.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.320+0000 7f659fdfb640 10 monclient: init 2024-04-04T21:02:31.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659fdfb640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659fdfb640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-04T21:02:31.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659fdfb640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65980a9b60 0x7f65981430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659fdfb640 10 monclient(hunting): picked mon.noname-b con 0x7f65980a9b60 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:31.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659fdfb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65981426f0 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:31.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659fdfb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f659813f810 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:31.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659fdfb640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65981435e0 con 0x7f65981426f0 2024-04-04T21:02:31.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659fdfb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65980b8c90 con 0x7f659813f810 2024-04-04T21:02:31.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659fdfb640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659fdfb640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65980a9f40 con 0x7f65980a9b60 2024-04-04T21:02:31.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659fdfb640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659fdfb640 10 monclient(hunting): authenticate will time out at 1400.588350s 2024-04-04T21:02:31.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659e398640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f659813f810 0x7f659813fc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:41960/0 (socket says 172.21.15.106:41960) 2024-04-04T21:02:31.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659e398640 1 -- 172.21.15.106:0/1507041139 learned_addr learned my addr 172.21.15.106:0/1507041139 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:31.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.321+0000 7f659d396640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f65981426f0 0x7f65980b28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:49600/0 (socket says 172.21.15.106:49600) 2024-04-04T21:02:31.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.324+0000 7f659cb95640 1 -- 172.21.15.106:0/1507041139 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778162616 0 0) 0x7f65981435e0 con 0x7f65981426f0 2024-04-04T21:02:31.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.324+0000 7f659cb95640 10 monclient(hunting): _negotiate 2024-04-04T21:02:31.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.324+0000 7f659cb95640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.324+0000 7f659cb95640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.324+0000 7f659cb95640 10 monclient(hunting): my global_id is 4359 2024-04-04T21:02:31.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.324+0000 7f659cb95640 1 -- 172.21.15.106:0/1507041139 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6584003520 con 0x7f65981426f0 2024-04-04T21:02:31.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 -- 172.21.15.106:0/1507041139 <== mon.0 v1:172.21.15.106:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1160026535 0 0) 0x7f6584003520 con 0x7f65981426f0 2024-04-04T21:02:31.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 -- 172.21.15.106:0/1507041139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65980a9b60 msgr2=0x7f65981430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:31.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 --2- 172.21.15.106:0/1507041139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65980a9b60 0x7f65981430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 -- 172.21.15.106:0/1507041139 >> v1:172.21.15.203:6789/0 conn(0x7f659813f810 legacy=0x7f659813fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 monclient: found mon.noname-a-legacy 2024-04-04T21:02:31.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.106:6789/0 2024-04-04T21:02:31.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 -- 172.21.15.106:0/1507041139 --> v1:172.21.15.106:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65980b23b0 con 0x7f65981426f0 2024-04-04T21:02:31.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient: _finish_auth 0 2024-04-04T21:02:31.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient: _check_auth_tickets 2024-04-04T21:02:31.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:31.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659fdfb640 5 monclient: authenticate success, global_id 4359 2024-04-04T21:02:31.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 -- 172.21.15.106:0/1507041139 <== mon.0 v1:172.21.15.106:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 141466616 0 0) 0x7f6588001670 con 0x7f65981426f0 2024-04-04T21:02:31.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659fdfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:31.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-04T21:02:31.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient: dump: 2024-04-04T21:02:31.330 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.330 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] but i'm connected to v1:172.21.15.106:6789/0, reconnecting 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 -- 172.21.15.106:0/1507041139 >> v1:172.21.15.106:6789/0 conn(0x7f65981426f0 legacy=0x7f65980b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 --2- 172.21.15.106:0/1507041139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65840039e0 0x7f6584003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient(hunting): picked mon.a con 0x7f65840039e0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:31.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 --2- 172.21.15.106:0/1507041139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65980a9b60 0x7f6584004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient(hunting): picked mon.c con 0x7f65980a9b60 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:31.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 --2- 172.21.15.106:0/1507041139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6584004e10 0x7f6584005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient(hunting): picked mon.b con 0x7f6584004e10 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:31.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 -- 172.21.15.106:0/1507041139 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65980a9f40 con 0x7f65840039e0 2024-04-04T21:02:31.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 -- 172.21.15.106:0/1507041139 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658400bcc0 con 0x7f6584004e10 2024-04-04T21:02:31.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 1 -- 172.21.15.106:0/1507041139 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658400bec0 con 0x7f65980a9b60 2024-04-04T21:02:31.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659cb95640 10 monclient(hunting): _finish_auth 0 2024-04-04T21:02:31.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.328+0000 7f659fdfb640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:31.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.330+0000 7f659e398640 1 --2- 172.21.15.106:0/1507041139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6584004e10 0x7f6584005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.330+0000 7f659e398640 10 monclient(hunting): get_auth_request con 0x7f6584004e10 auth_method 0 2024-04-04T21:02:31.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.330+0000 7f659e398640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.330+0000 7f659e398640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.330+0000 7f659e398640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-04T21:02:31.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.330+0000 7f659e398640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.330+0000 7f659e398640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.330+0000 7f659e398640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-04T21:02:31.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.332+0000 7f659e398640 10 monclient(hunting): handle_auth_done global_id 4359 payload 293 2024-04-04T21:02:31.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.332+0000 7f659db97640 1 --2- 172.21.15.106:0/1507041139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65840039e0 0x7f6584003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.332+0000 7f659e398640 1 -- 172.21.15.106:0/1507041139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65980a9b60 msgr2=0x7f6584004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:31.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.332+0000 7f659e398640 1 --2- 172.21.15.106:0/1507041139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65980a9b60 0x7f6584004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.332+0000 7f659e398640 1 -- 172.21.15.106:0/1507041139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65840039e0 msgr2=0x7f6584003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:31.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.332+0000 7f659e398640 1 --2- 172.21.15.106:0/1507041139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65840039e0 0x7f6584003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.332+0000 7f659e398640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.332+0000 7f659e398640 1 monclient: found mon.b 2024-04-04T21:02:31.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.332+0000 7f659e398640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.332+0000 7f659e398640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:31.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.332+0000 7f659e398640 1 -- 172.21.15.106:0/1507041139 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6584003520 con 0x7f6584004e10 2024-04-04T21:02:31.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.332+0000 7f659e398640 10 monclient: _finish_auth 0 2024-04-04T21:02:31.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.332+0000 7f659e398640 10 monclient: _check_auth_tickets 2024-04-04T21:02:31.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.332+0000 7f659e398640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:31.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.334+0000 7f659e398640 1 --2- 172.21.15.106:0/1507041139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6584004e10 0x7f6584005290 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f659000c190 tx=0x7f6584005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc97b38ba4be5fb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659cb95640 1 -- 172.21.15.106:0/1507041139 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6590001ee0 con 0x7f6584004e10 2024-04-04T21:02:31.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659cb95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659cb95640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659cb95640 10 monclient: dump: 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659fdfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659cb95640 1 -- 172.21.15.106:0/1507041139 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6590030070 con 0x7f6584004e10 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659cb95640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659fdfb640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659fdfb640 4 set_mon_vals no callback set 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659cb95640 1 -- 172.21.15.106:0/1507041139 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659002cc90 con 0x7f6584004e10 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659cb95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659cb95640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659cb95640 10 monclient: dump: 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.340 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659fdfb640 10 monclient: shutdown 2024-04-04T21:02:31.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659fdfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:31.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659fdfb640 1 -- 172.21.15.106:0/1507041139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6584004e10 msgr2=0x7f6584005290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659fdfb640 1 --2- 172.21.15.106:0/1507041139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6584004e10 0x7f6584005290 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f659000c190 tx=0x7f6584005830 comp rx=0 tx=0).stop 2024-04-04T21:02:31.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659e398640 1 -- 172.21.15.106:0/1507041139 reap_dead start 2024-04-04T21:02:31.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659fdfb640 1 -- 172.21.15.106:0/1507041139 shutdown_connections 2024-04-04T21:02:31.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.338+0000 7f659fdfb640 1 -- 172.21.15.106:0/1507041139 >> 172.21.15.106:0/1507041139 conn(0x7f65980b7ec0 msgr2=0x7f65980b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:31.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.339+0000 7f659fdfb640 1 -- 172.21.15.106:0/1507041139 shutdown_connections 2024-04-04T21:02:31.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.339+0000 7f659fdfb640 1 -- 172.21.15.106:0/1507041139 wait complete. 2024-04-04T21:02:31.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.339+0000 7f659fdfb640 10 monclient: build_initial_monmap 2024-04-04T21:02:31.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.339+0000 7f659fdfb640 10 monclient: monmap: 2024-04-04T21:02:31.345 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:31.345 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:31.346 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:31.346 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:31.346 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:31.346 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.346 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:31.346 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:31.347 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:31.347 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.339+0000 7f659fdfb640 1 librados: starting msgr at 2024-04-04T21:02:31.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.339+0000 7f659fdfb640 1 librados: starting objecter 2024-04-04T21:02:31.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.339+0000 7f659fdfb640 1 Processor -- start 2024-04-04T21:02:31.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 1 -- start start 2024-04-04T21:02:31.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 1 librados: setting wanted keys 2024-04-04T21:02:31.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 1 librados: calling monclient init 2024-04-04T21:02:31.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 10 monclient: init 2024-04-04T21:02:31.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-04T21:02:31.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6584004e10 0x7f65980b6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 10 monclient(hunting): picked mon.noname-b con 0x7f6584004e10 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:31.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6598145530 0x7f6598145970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 10 monclient(hunting): picked mon.noname-c con 0x7f6598145530 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:31.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65981db390 0x7f65981db810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 10 monclient(hunting): picked mon.noname-a con 0x7f65981db390 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:31.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6598142c10 con 0x7f65981db390 2024-04-04T21:02:31.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6598143370 con 0x7f6584004e10 2024-04-04T21:02:31.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6598146fd0 con 0x7f6598145530 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.340+0000 7f659fdfb640 10 monclient(hunting): authenticate will time out at 1400.607753s 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659e398640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65981db390 0x7f65981db810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659db97640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6584004e10 0x7f65980b6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659e398640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65981db390 0x7f65981db810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35124/0 (socket says 172.21.15.106:35124) 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659e398640 1 -- 172.21.15.106:0/2375176186 learned_addr learned my addr 172.21.15.106:0/2375176186 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659e398640 10 monclient(hunting): get_auth_request con 0x7f65981db390 auth_method 0 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659e398640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659e398640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659e398640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659db97640 10 monclient(hunting): get_auth_request con 0x7f6584004e10 auth_method 0 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659db97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659db97640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659db97640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659e398640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659e398640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659e398640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659db97640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659db97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.341+0000 7f659db97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-04T21:02:31.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.344+0000 7f659db97640 10 monclient(hunting): handle_auth_done global_id 4405 payload 824 2024-04-04T21:02:31.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.344+0000 7f659db97640 1 -- 172.21.15.106:0/2375176186 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6598145530 msgr2=0x7f6598145970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.344+0000 7f659db97640 1 --2- 172.21.15.106:0/2375176186 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6598145530 0x7f6598145970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.344+0000 7f659db97640 1 -- 172.21.15.106:0/2375176186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65981db390 msgr2=0x7f65981db810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.344+0000 7f659db97640 1 --2- 172.21.15.106:0/2375176186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65981db390 0x7f65981db810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.344+0000 7f659db97640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.344+0000 7f659db97640 1 monclient: found mon.noname-b 2024-04-04T21:02:31.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.344+0000 7f659db97640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.344+0000 7f659db97640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:31.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.344+0000 7f659db97640 1 -- 172.21.15.106:0/2375176186 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65981dbd50 con 0x7f6584004e10 2024-04-04T21:02:31.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.346+0000 7f659db97640 1 --2- 172.21.15.106:0/2375176186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6584004e10 0x7f65980b6d70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f659400c570 tx=0x7f659400ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.346+0000 7f658e7fc640 10 client.?.objecter ms_handle_connect 0x7f6584004e10 2024-04-04T21:02:31.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.346+0000 7f658e7fc640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:31.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f658e7fc640 1 -- 172.21.15.106:0/2375176186 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594015070 con 0x7f6584004e10 2024-04-04T21:02:31.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f658e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f658e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f658e7fc640 10 monclient: dump: 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f658e7fc640 10 monclient: _finish_auth 0 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f658e7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:02:31.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f658e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:31.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f658e7fc640 1 -- 172.21.15.106:0/2375176186 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6594011800 con 0x7f6584004e10 2024-04-04T21:02:31.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f658e7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:31.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f658e7fc640 1 -- 172.21.15.106:0/2375176186 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594011bb0 con 0x7f6584004e10 2024-04-04T21:02:31.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f659fdfb640 5 monclient: authenticate success, global_id 4405 2024-04-04T21:02:31.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f659fdfb640 10 monclient: _renew_subs 2024-04-04T21:02:31.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f659fdfb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:31.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f659fdfb640 1 -- 172.21.15.106:0/2375176186 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65981dbfe0 con 0x7f6584004e10 2024-04-04T21:02:31.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f658e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f658e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:31.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f658e7fc640 10 monclient: dump: 2024-04-04T21:02:31.356 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.356 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.356 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.356 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f658f7fe640 4 set_mon_vals no callback set 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f659fdfb640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f659fdfb640 10 monclient: _renew_subs 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f659fdfb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f659fdfb640 1 -- 172.21.15.106:0/2375176186 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65981d7fc0 con 0x7f6584004e10 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.350+0000 7f659fdfb640 1 librados: init done 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.351+0000 7f659fdfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.351+0000 7f659fdfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.351+0000 7f659fdfb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:31.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.351+0000 7f659fdfb640 1 -- 172.21.15.106:0/2375176186 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65980ac250 con 0x7f6584004e10 2024-04-04T21:02:31.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.368+0000 7f658e7fc640 1 -- 172.21.15.106:0/2375176186 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6594018070 con 0x7f6584004e10 2024-04-04T21:02:31.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.369+0000 7f658e7fc640 1 --2- 172.21.15.106:0/2375176186 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f657c06cee0 0x7f657c06f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.369+0000 7f659d396640 1 --2- 172.21.15.106:0/2375176186 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f657c06cee0 0x7f657c06f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.369+0000 7f659d396640 10 monclient: get_auth_request con 0x7f657c06cee0 auth_method 0 2024-04-04T21:02:31.370 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.369+0000 7f659d396640 1 --2- 172.21.15.106:0/2375176186 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f657c06cee0 0x7f657c06f3a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6588001eb0 tx=0x7f6588001600 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.370 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.369+0000 7f658e7fc640 10 client.4405.objecter ms_handle_connect 0x7f657c06cee0 2024-04-04T21:02:31.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.370+0000 7f658e7fc640 1 -- 172.21.15.106:0/2375176186 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6594056b60 con 0x7f6584004e10 2024-04-04T21:02:31.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.370+0000 7f658e7fc640 10 client.4405.objecter ms_dispatch 0x7f6598000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:31.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.370+0000 7f658e7fc640 3 client.4405.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:31.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.370+0000 7f658e7fc640 3 client.4405.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:31.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.370+0000 7f658e7fc640 20 client.4405.objecter dump_active .. 0 homeless 2024-04-04T21:02:31.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-04T21:02:31.422 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-04T21:02:31.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.433+0000 7f6574ff9640 1 -- 172.21.15.106:0/2383310870 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6568007220 con 0x7f65780b0b10 2024-04-04T21:02:31.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.433+0000 7f6574ff9640 1 --2- 172.21.15.106:0/2383310870 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f656406ce60 0x7f656406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.434+0000 7f6577fff640 1 --2- 172.21.15.106:0/2383310870 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f656406ce60 0x7f656406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.434+0000 7f6577fff640 10 monclient: get_auth_request con 0x7f656406ce60 auth_method 0 2024-04-04T21:02:31.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.434+0000 7f6574ff9640 10 client.4337.objecter ms_handle_connect 0x7f656406ce60 2024-04-04T21:02:31.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.434+0000 7f6577fff640 1 --2- 172.21.15.106:0/2383310870 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f656406ce60 0x7f656406f320 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f656c0011e0 tx=0x7f656c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.452+0000 7f6574ff9640 1 -- 172.21.15.106:0/2383310870 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6568056360 con 0x7f65780b0b10 2024-04-04T21:02:31.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.452+0000 7f6574ff9640 10 client.4337.objecter ms_dispatch 0x7f6578000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:31.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.452+0000 7f6574ff9640 3 client.4337.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:31.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.452+0000 7f6574ff9640 3 client.4337.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:31.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.452+0000 7f6574ff9640 20 client.4337.objecter dump_active .. 0 homeless 2024-04-04T21:02:31.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.452+0000 7f6574ff9640 1 -- 172.21.15.106:0/2383310870 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f65680582c0 con 0x7f65780b0b10 2024-04-04T21:02:31.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.452+0000 7f6574ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.452+0000 7f6574ff9640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:31.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.466+0000 7f5bd35c5640 10 monclient: get_monmap_and_config 2024-04-04T21:02:31.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.466+0000 7f5bd35c5640 10 monclient: build_initial_monmap 2024-04-04T21:02:31.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.466+0000 7f5bd35c5640 10 monclient: monmap: 2024-04-04T21:02:31.466 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:31.467 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.467 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:31.467449+0000 2024-04-04T21:02:31.467 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:31.467449+0000 2024-04-04T21:02:31.467 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:31.467 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.467 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:31.467 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:31.467 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:31.467 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:31.467 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:31.467 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 1 Processor -- start 2024-04-04T21:02:31.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 1 -- start start 2024-04-04T21:02:31.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 10 monclient: init 2024-04-04T21:02:31.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-04T21:02:31.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5bcc1442f0 0x7f5bcc0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 10 monclient(hunting): picked mon.noname-b con 0x7f5bcc1442f0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:31.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5bcc14a060 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:31.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5bcc147150 0x7f5bcc1475e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 10 monclient(hunting): picked mon.noname-c con 0x7f5bcc147150 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:31.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bcc06f6a0 con 0x7f5bcc14a060 2024-04-04T21:02:31.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bcc06e130 con 0x7f5bcc147150 2024-04-04T21:02:31.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc146c30 con 0x7f5bcc1442f0 2024-04-04T21:02:31.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.467+0000 7f5bd35c5640 10 monclient(hunting): authenticate will time out at 1400.735116s 2024-04-04T21:02:31.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.468+0000 7f5bd1361640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5bcc1442f0 0x7f5bcc0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.469+0000 7f5bd1361640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5bcc1442f0 0x7f5bcc0b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:33130/0 (socket says 172.21.15.106:33130) 2024-04-04T21:02:31.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.469+0000 7f5bd1361640 1 -- 172.21.15.106:0/3198270141 learned_addr learned my addr 172.21.15.106:0/3198270141 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:31.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.469+0000 7f5bd1361640 10 monclient(hunting): get_auth_request con 0x7f5bcc1442f0 auth_method 0 2024-04-04T21:02:31.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.469+0000 7f5bd1361640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.469+0000 7f5bd1361640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.469+0000 7f5bd1361640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.470+0000 7f5bd1361640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.470+0000 7f5bd1361640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.470+0000 7f5bd1361640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.472+0000 7f5bd1361640 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2024-04-04T21:02:31.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.472+0000 7f5bd1361640 1 -- 172.21.15.106:0/3198270141 >> v2:172.21.15.203:3300/0 conn(0x7f5bcc147150 msgr2=0x7f5bcc1475e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:31.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.472+0000 7f5bd1361640 1 --2- 172.21.15.106:0/3198270141 >> v2:172.21.15.203:3300/0 conn(0x7f5bcc147150 0x7f5bcc1475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.472+0000 7f5bd1361640 1 -- 172.21.15.106:0/3198270141 >> v1:172.21.15.203:6789/0 conn(0x7f5bcc14a060 legacy=0x7f5bcc0b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:31.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.472+0000 7f5bd1361640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.472+0000 7f5bd1361640 1 monclient: found mon.noname-b 2024-04-04T21:02:31.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.472+0000 7f5bd1361640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.472+0000 7f5bd1361640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.472+0000 7f5bd1361640 1 -- 172.21.15.106:0/3198270141 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc152400 con 0x7f5bcc1442f0 2024-04-04T21:02:31.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.475+0000 7f5bd1361640 1 --2- 172.21.15.106:0/3198270141 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5bcc1442f0 0x7f5bcc0b3f80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009870 tx=0x7f5bc802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75a65ddda4213b73 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bc3fff640 1 -- 172.21.15.106:0/3198270141 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc80317c0 con 0x7f5bcc1442f0 2024-04-04T21:02:31.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bc3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:31.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bc3fff640 10 monclient: dump: 2024-04-04T21:02:31.481 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.481 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.481 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.481 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.481 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.481 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.481 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.482 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.482 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.482 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bc3fff640 10 monclient: _finish_auth 0 2024-04-04T21:02:31.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bc3fff640 10 monclient: _check_auth_tickets 2024-04-04T21:02:31.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:31.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bc3fff640 1 -- 172.21.15.106:0/3198270141 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc8037070 con 0x7f5bcc1442f0 2024-04-04T21:02:31.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bc3fff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:31.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bc3fff640 1 -- 172.21.15.106:0/3198270141 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8031b80 con 0x7f5bcc1442f0 2024-04-04T21:02:31.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:31.482 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bc3fff640 10 monclient: dump: 2024-04-04T21:02:31.482 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.482 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.483 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.483 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.483 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.483 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.483 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.483 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.483 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.483 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.483 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bd35c5640 5 monclient: authenticate success, global_id 4340 2024-04-04T21:02:31.483 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bd35c5640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:31.483 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.480+0000 7f5bd35c5640 4 set_mon_vals no callback set 2024-04-04T21:02:31.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.481+0000 7f5bd35c5640 10 monclient: shutdown 2024-04-04T21:02:31.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.481+0000 7f5bd35c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:31.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.481+0000 7f5bd35c5640 1 -- 172.21.15.106:0/3198270141 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5bcc1442f0 msgr2=0x7f5bcc0b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.481+0000 7f5bd35c5640 1 --2- 172.21.15.106:0/3198270141 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5bcc1442f0 0x7f5bcc0b3f80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009870 tx=0x7f5bc802b860 comp rx=0 tx=0).stop 2024-04-04T21:02:31.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.481+0000 7f5bd35c5640 1 -- 172.21.15.106:0/3198270141 shutdown_connections 2024-04-04T21:02:31.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.481+0000 7f5bd35c5640 1 --2- 172.21.15.106:0/3198270141 >> v2:172.21.15.203:3300/0 conn(0x7f5bcc147150 0x7f5bcc1475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.481+0000 7f5bd35c5640 1 --2- 172.21.15.106:0/3198270141 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5bcc1442f0 0x7f5bcc0b3f80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.481+0000 7f5bd35c5640 1 -- 172.21.15.106:0/3198270141 >> 172.21.15.106:0/3198270141 conn(0x7f5bcc0b8030 msgr2=0x7f5bcc0b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:31.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.481+0000 7f5bd35c5640 1 -- 172.21.15.106:0/3198270141 shutdown_connections 2024-04-04T21:02:31.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.481+0000 7f5bd35c5640 1 -- 172.21.15.106:0/3198270141 wait complete. 2024-04-04T21:02:31.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.482+0000 7f5bd35c5640 10 monclient: build_initial_monmap 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.482+0000 7f5bd35c5640 10 monclient: monmap: 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.483+0000 7f5bd35c5640 1 librados: starting msgr at 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.483+0000 7f5bd35c5640 1 librados: starting objecter 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.483+0000 7f5bd35c5640 1 Processor -- start 2024-04-04T21:02:31.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.483+0000 7f5bd35c5640 1 -- start start 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.483+0000 7f5bd35c5640 1 librados: setting wanted keys 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.483+0000 7f5bd35c5640 1 librados: calling monclient init 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.483+0000 7f5bd35c5640 10 monclient: init 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5bcc1442f0 0x7f5bcc143790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 10 monclient(hunting): picked mon.noname-a con 0x7f5bcc1442f0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc147150 0x7f5bcc141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 10 monclient(hunting): picked mon.noname-b con 0x7f5bcc147150 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5bcc14a060 0x7f5bcc141570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 10 monclient(hunting): picked mon.noname-c con 0x7f5bcc14a060 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc151780 con 0x7f5bcc1442f0 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc151600 con 0x7f5bcc147150 2024-04-04T21:02:31.488 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc151900 con 0x7f5bcc14a060 2024-04-04T21:02:31.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.484+0000 7f5bd35c5640 10 monclient(hunting): authenticate will time out at 1400.751944s 2024-04-04T21:02:31.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.487+0000 7f5bd1361640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5bcc1442f0 0x7f5bcc143790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.487+0000 7f5bd1361640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5bcc1442f0 0x7f5bcc143790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35136/0 (socket says 172.21.15.106:35136) 2024-04-04T21:02:31.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.487+0000 7f5bd1361640 1 -- 172.21.15.106:0/2769814709 learned_addr learned my addr 172.21.15.106:0/2769814709 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:31.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.487+0000 7f5bd1361640 10 monclient(hunting): get_auth_request con 0x7f5bcc1442f0 auth_method 0 2024-04-04T21:02:31.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.487+0000 7f5bd1361640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.487+0000 7f5bd1361640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.487+0000 7f5bd1361640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.489+0000 7f5bd1361640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.489+0000 7f5bd1361640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.489+0000 7f5bd1361640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.492+0000 7f5bd1b62640 1 --2- 172.21.15.106:0/2769814709 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5bcc14a060 0x7f5bcc141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.493+0000 7f5bd1b62640 10 monclient(hunting): get_auth_request con 0x7f5bcc14a060 auth_method 0 2024-04-04T21:02:31.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.493+0000 7f5bd1b62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.493+0000 7f5bd1b62640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.493+0000 7f5bd1b62640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.494 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.494+0000 7f5bd1b62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.494 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.494+0000 7f5bd1b62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.494 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.494+0000 7f5bd1b62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.495+0000 7f5bd1361640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2024-04-04T21:02:31.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.495+0000 7f5bd1361640 1 -- 172.21.15.106:0/2769814709 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5bcc14a060 msgr2=0x7f5bcc141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.495+0000 7f5bd1361640 1 --2- 172.21.15.106:0/2769814709 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5bcc14a060 0x7f5bcc141570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.495+0000 7f5bd1361640 1 -- 172.21.15.106:0/2769814709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc147150 msgr2=0x7f5bcc141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.495+0000 7f5bd1361640 1 --2- 172.21.15.106:0/2769814709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc147150 0x7f5bcc141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.495+0000 7f5bd1361640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.495+0000 7f5bd1361640 1 monclient: found mon.noname-a 2024-04-04T21:02:31.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.495+0000 7f5bd1361640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.495+0000 7f5bd1361640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:31.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.495+0000 7f5bd1361640 1 -- 172.21.15.106:0/2769814709 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc141ab0 con 0x7f5bcc1442f0 2024-04-04T21:02:31.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.501+0000 7f5bd1361640 1 --2- 172.21.15.106:0/2769814709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5bcc1442f0 0x7f5bcc143790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009870 tx=0x7f5bc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.501+0000 7f5bc1ffb640 10 client.?.objecter ms_handle_connect 0x7f5bcc1442f0 2024-04-04T21:02:31.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.501+0000 7f5bc1ffb640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:31.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.503+0000 7ff4117a5640 10 monclient: get_monmap_and_config 2024-04-04T21:02:31.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.503+0000 7ff4117a5640 10 monclient: build_initial_monmap 2024-04-04T21:02:31.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.503+0000 7ff4117a5640 10 monclient: monmap: 2024-04-04T21:02:31.504 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:31.504 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.504 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:31.504989+0000 2024-04-04T21:02:31.504 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:31.504989+0000 2024-04-04T21:02:31.504 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:31.504 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.505 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:31.505 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:31.505 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:31.505 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:31.505 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:31.505 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.504+0000 7ff4117a5640 1 Processor -- start 2024-04-04T21:02:31.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.504+0000 7ff4117a5640 1 -- start start 2024-04-04T21:02:31.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.504+0000 7ff4117a5640 10 monclient: init 2024-04-04T21:02:31.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.505+0000 7ff4117a5640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.505+0000 7ff4117a5640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-04-04T21:02:31.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.505+0000 7ff4117a5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff40c070e50 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:31.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.505+0000 7ff4117a5640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c073110 0x7ff40c071230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.505+0000 7ff4117a5640 10 monclient(hunting): picked mon.noname-b con 0x7ff40c073110 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:31.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.505+0000 7ff4117a5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff40c143eb0 0x7ff40c1442e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.505+0000 7ff4117a5640 10 monclient(hunting): picked mon.noname-c con 0x7ff40c143eb0 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:31.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.505+0000 7ff4117a5640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40c144820 con 0x7ff40c070e50 2024-04-04T21:02:31.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.505+0000 7ff4117a5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.505+0000 7ff4117a5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40c143a20 con 0x7ff40c143eb0 2024-04-04T21:02:31.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.505+0000 7ff4117a5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.505+0000 7ff4117a5640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c13df50 con 0x7ff40c073110 2024-04-04T21:02:31.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.505+0000 7ff4117a5640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.505+0000 7ff4117a5640 10 monclient(hunting): authenticate will time out at 1400.772808s 2024-04-04T21:02:31.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.549+0000 7ff403fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c073110 0x7ff40c071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.553+0000 7ff403fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c073110 0x7ff40c071230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:33154/0 (socket says 172.21.15.106:33154) 2024-04-04T21:02:31.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.553+0000 7ff403fff640 1 -- 172.21.15.106:0/3122554073 learned_addr learned my addr 172.21.15.106:0/3122554073 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:31.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.553+0000 7ff403fff640 10 monclient(hunting): get_auth_request con 0x7ff40c073110 auth_method 0 2024-04-04T21:02:31.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.553+0000 7ff403fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.553+0000 7ff403fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.553+0000 7ff403fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.554+0000 7ff403fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.554+0000 7ff403fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.554+0000 7ff403fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.556+0000 7ff403fff640 10 monclient(hunting): handle_auth_done global_id 4346 payload 274 2024-04-04T21:02:31.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.556+0000 7ff403fff640 1 -- 172.21.15.106:0/3122554073 >> v2:172.21.15.203:3300/0 conn(0x7ff40c143eb0 msgr2=0x7ff40c1442e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:31.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.556+0000 7ff403fff640 1 --2- 172.21.15.106:0/3122554073 >> v2:172.21.15.203:3300/0 conn(0x7ff40c143eb0 0x7ff40c1442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.556+0000 7ff403fff640 1 -- 172.21.15.106:0/3122554073 >> v1:172.21.15.106:6789/0 conn(0x7ff40c070e50 legacy=0x7ff40c13d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:31.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.556+0000 7ff403fff640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.556+0000 7ff403fff640 1 monclient: found mon.noname-b 2024-04-04T21:02:31.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.556+0000 7ff403fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.556+0000 7ff403fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.556+0000 7ff403fff640 1 -- 172.21.15.106:0/3122554073 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40c0b0dd0 con 0x7ff40c073110 2024-04-04T21:02:31.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.559+0000 7fb30feb7640 10 monclient: get_monmap_and_config 2024-04-04T21:02:31.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.559+0000 7fb30feb7640 10 monclient: build_initial_monmap 2024-04-04T21:02:31.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.559+0000 7fb30feb7640 10 monclient: monmap: 2024-04-04T21:02:31.559 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:31.559 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.559 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:31.560086+0000 2024-04-04T21:02:31.559 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:31.560086+0000 2024-04-04T21:02:31.559 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:31.560 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.560 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:31.560 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:31.560 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:31.560 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:31.560 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:31.560 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.559+0000 7ff403fff640 1 --2- 172.21.15.106:0/3122554073 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c073110 0x7ff40c071230 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4009870 tx=0x7ff3f402fbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb6c95043da834d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.559+0000 7fb30feb7640 1 Processor -- start 2024-04-04T21:02:31.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.559+0000 7fb30feb7640 1 -- start start 2024-04-04T21:02:31.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.559+0000 7fb30feb7640 10 monclient: init 2024-04-04T21:02:31.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.560+0000 7fb30feb7640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.560+0000 7fb30feb7640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-04T21:02:31.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.560+0000 7fb30feb7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3080b08a0 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:31.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.560+0000 7fb30feb7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb3080b0c80 0x7fb3080725f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.560+0000 7fb30feb7640 10 monclient(hunting): picked mon.noname-c con 0x7fb3080b0c80 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:31.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.560+0000 7fb30feb7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb308070fc0 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:31.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.560+0000 7fb30feb7640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb308072240 con 0x7fb3080b08a0 2024-04-04T21:02:31.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.560+0000 7fb30feb7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3080b14b0 con 0x7fb308070fc0 2024-04-04T21:02:31.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.560+0000 7fb30feb7640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.560+0000 7fb30feb7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb308072d10 con 0x7fb3080b0c80 2024-04-04T21:02:31.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.560+0000 7fb30feb7640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.560+0000 7fb30feb7640 10 monclient(hunting): authenticate will time out at 1400.827985s 2024-04-04T21:02:31.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff40a7fc640 1 -- 172.21.15.106:0/3122554073 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f403d070 con 0x7ff40c073110 2024-04-04T21:02:31.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff40a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff40a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff40a7fc640 10 monclient: dump: 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff40a7fc640 10 monclient: _finish_auth 0 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff40a7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff40a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff40a7fc640 1 -- 172.21.15.106:0/3122554073 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f40309a0 con 0x7ff40c073110 2024-04-04T21:02:31.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff40a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff40a7fc640 1 -- 172.21.15.106:0/3122554073 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4030d70 con 0x7ff40c073110 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff40a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff40a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff40a7fc640 10 monclient: dump: 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff4117a5640 5 monclient: authenticate success, global_id 4346 2024-04-04T21:02:31.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff4117a5640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:31.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.563+0000 7ff4117a5640 4 set_mon_vals no callback set 2024-04-04T21:02:31.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.564+0000 7ff4117a5640 10 monclient: shutdown 2024-04-04T21:02:31.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.564+0000 7ff4117a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.564+0000 7ff4117a5640 1 -- 172.21.15.106:0/3122554073 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c073110 msgr2=0x7ff40c071230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.564+0000 7ff4117a5640 1 --2- 172.21.15.106:0/3122554073 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c073110 0x7ff40c071230 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4009870 tx=0x7ff3f402fbf0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.564+0000 7ff4117a5640 1 -- 172.21.15.106:0/3122554073 shutdown_connections 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.564+0000 7ff4117a5640 1 --2- 172.21.15.106:0/3122554073 >> v2:172.21.15.203:3300/0 conn(0x7ff40c143eb0 0x7ff40c1442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.564+0000 7ff4117a5640 1 --2- 172.21.15.106:0/3122554073 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c073110 0x7ff40c071230 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.564+0000 7ff4117a5640 1 -- 172.21.15.106:0/3122554073 >> 172.21.15.106:0/3122554073 conn(0x7ff40c0b62c0 msgr2=0x7ff40c0b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.564+0000 7ff4117a5640 1 -- 172.21.15.106:0/3122554073 shutdown_connections 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.564+0000 7ff4117a5640 1 -- 172.21.15.106:0/3122554073 wait complete. 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.565+0000 7ff4117a5640 10 monclient: build_initial_monmap 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.565+0000 7ff4117a5640 10 monclient: monmap: 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:31.570 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.566+0000 7ff4117a5640 1 librados: starting msgr at 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.566+0000 7ff4117a5640 1 librados: starting objecter 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.566+0000 7ff4117a5640 1 Processor -- start 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.566+0000 7ff4117a5640 1 -- start start 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.566+0000 7ff4117a5640 1 librados: setting wanted keys 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.566+0000 7ff4117a5640 1 librados: calling monclient init 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.566+0000 7ff4117a5640 10 monclient: init 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-04T21:02:31.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff40c070e50 0x7ff40c13cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 10 monclient(hunting): picked mon.noname-a con 0x7ff40c070e50 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c143eb0 0x7ff40c1d3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 10 monclient(hunting): picked mon.noname-b con 0x7ff40c143eb0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c1d3a70 0x7ff40c140810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 10 monclient(hunting): picked mon.noname-c con 0x7ff40c1d3a70 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c06be20 con 0x7ff40c070e50 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c0723f0 con 0x7ff40c143eb0 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c06bfa0 con 0x7ff40c1d3a70 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff4117a5640 10 monclient(hunting): authenticate will time out at 1400.834573s 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.567+0000 7ff40b7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c1d3a70 0x7ff40c140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.568+0000 7ff40b7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c1d3a70 0x7ff40c140810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:33156/0 (socket says 172.21.15.106:33156) 2024-04-04T21:02:31.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.568+0000 7ff40b7fe640 1 -- 172.21.15.106:0/239103095 learned_addr learned my addr 172.21.15.106:0/239103095 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:31.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.568+0000 7ff40b7fe640 10 monclient(hunting): get_auth_request con 0x7ff40c1d3a70 auth_method 0 2024-04-04T21:02:31.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.568+0000 7ff40b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.568+0000 7ff40b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.568+0000 7ff40b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.568+0000 7ff40b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.568+0000 7ff40b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.568+0000 7ff40b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7ff40b7fe640 10 monclient(hunting): handle_auth_done global_id 4349 payload 824 2024-04-04T21:02:31.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7ff40b7fe640 1 -- 172.21.15.106:0/239103095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c143eb0 msgr2=0x7ff40c1d3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7ff40b7fe640 1 --2- 172.21.15.106:0/239103095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c143eb0 0x7ff40c1d3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7ff40b7fe640 1 -- 172.21.15.106:0/239103095 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff40c070e50 msgr2=0x7ff40c13cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7ff40b7fe640 1 --2- 172.21.15.106:0/239103095 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff40c070e50 0x7ff40c13cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7ff40b7fe640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7ff40b7fe640 1 monclient: found mon.noname-c 2024-04-04T21:02:31.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7ff40b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7ff40b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7ff40b7fe640 1 -- 172.21.15.106:0/239103095 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40c140d50 con 0x7ff40c1d3a70 2024-04-04T21:02:31.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7f5bc1ffb640 1 -- 172.21.15.106:0/2769814709 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8032860 con 0x7f5bcc1442f0 2024-04-04T21:02:31.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7f5bc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7f5bc1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:31.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7f5bc1ffb640 10 monclient: dump: 2024-04-04T21:02:31.576 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.576 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.576 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.576 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.576 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.576 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.576 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.576 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.576 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.577 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7f5bc1ffb640 10 monclient: _finish_auth 0 2024-04-04T21:02:31.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7f5bc1ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:02:31.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.571+0000 7f5bc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:31.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.572+0000 7f5bd35c5640 5 monclient: authenticate success, global_id 4365 2024-04-04T21:02:31.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.572+0000 7f5bd35c5640 10 monclient: _renew_subs 2024-04-04T21:02:31.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.572+0000 7f5bd35c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:31.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.572+0000 7f5bd35c5640 1 -- 172.21.15.106:0/2769814709 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bcc14c900 con 0x7f5bcc1442f0 2024-04-04T21:02:31.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.572+0000 7f5bd35c5640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:31.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.572+0000 7f5bd35c5640 10 monclient: _renew_subs 2024-04-04T21:02:31.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.572+0000 7f5bd35c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:31.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.572+0000 7f5bd35c5640 1 -- 172.21.15.106:0/2769814709 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bcc14b850 con 0x7f5bcc1442f0 2024-04-04T21:02:31.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.572+0000 7f5bd35c5640 1 librados: init done 2024-04-04T21:02:31.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.572+0000 7fb30dc53640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fb3080b08a0 0x7fb30813fd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:49616/0 (socket says 172.21.15.106:49616) 2024-04-04T21:02:31.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.572+0000 7fb30dc53640 1 -- 172.21.15.106:0/2123984874 learned_addr learned my addr 172.21.15.106:0/2123984874 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:31.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.573+0000 7f5bd35c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.573+0000 7f5bd35c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.573+0000 7f5bd35c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:31.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.573+0000 7f5bd35c5640 1 -- 172.21.15.106:0/2769814709 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bcc04f050 con 0x7f5bcc1442f0 2024-04-04T21:02:31.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.574+0000 7ff40b7fe640 1 --2- 172.21.15.106:0/239103095 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c1d3a70 0x7ff40c140810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff40400b970 tx=0x7ff40400be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.574+0000 7ff4037fe640 10 client.?.objecter ms_handle_connect 0x7ff40c1d3a70 2024-04-04T21:02:31.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.574+0000 7ff4037fe640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:31.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.575+0000 7f5bc1ffb640 1 -- 172.21.15.106:0/2769814709 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc8037020 con 0x7f5bcc1442f0 2024-04-04T21:02:31.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.575+0000 7f5bc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:31.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.575+0000 7f5bc1ffb640 1 -- 172.21.15.106:0/2769814709 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8032c00 con 0x7f5bcc1442f0 2024-04-04T21:02:31.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.575+0000 7f5bc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.575+0000 7f5bc1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:31.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.575+0000 7f5bc1ffb640 10 monclient: dump: 2024-04-04T21:02:31.580 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.580 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.580 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.580 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.580 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.580 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.580 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.580 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.580 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.580 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.575+0000 7f5bc37fe640 4 set_mon_vals no callback set 2024-04-04T21:02:31.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4037fe640 1 -- 172.21.15.106:0/239103095 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff404019070 con 0x7ff40c1d3a70 2024-04-04T21:02:31.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:31.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4037fe640 10 monclient: dump: 2024-04-04T21:02:31.582 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.582 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.582 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4037fe640 10 monclient: _finish_auth 0 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4037fe640 10 monclient: _check_auth_tickets 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4037fe640 1 -- 172.21.15.106:0/239103095 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40400cb60 con 0x7ff40c1d3a70 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4037fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4037fe640 1 -- 172.21.15.106:0/239103095 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff404014440 con 0x7ff40c1d3a70 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4037fe640 10 monclient: dump: 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff409ffb640 4 set_mon_vals no callback set 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4117a5640 5 monclient: authenticate success, global_id 4349 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4117a5640 10 monclient: _renew_subs 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4117a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4117a5640 1 -- 172.21.15.106:0/239103095 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40c141040 con 0x7ff40c1d3a70 2024-04-04T21:02:31.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4117a5640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:31.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4117a5640 10 monclient: _renew_subs 2024-04-04T21:02:31.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4117a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4117a5640 1 -- 172.21.15.106:0/239103095 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40c1d6c50 con 0x7ff40c1d3a70 2024-04-04T21:02:31.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.580+0000 7ff4117a5640 1 librados: init done 2024-04-04T21:02:31.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.582+0000 7ff4117a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.582+0000 7ff4117a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.582+0000 7ff4117a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.585 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.582+0000 7ff4117a5640 1 -- 172.21.15.106:0/239103095 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff40c0b5cc0 con 0x7ff40c1d3a70 2024-04-04T21:02:31.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.600+0000 7ff4037fe640 1 -- 172.21.15.106:0/239103095 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff40401c070 con 0x7ff40c1d3a70 2024-04-04T21:02:31.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.601+0000 7ff4037fe640 1 --2- 172.21.15.106:0/239103095 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff3ec06ce60 0x7ff3ec06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.601+0000 7ff40affd640 1 --2- 172.21.15.106:0/239103095 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff3ec06ce60 0x7ff3ec06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.601+0000 7ff40affd640 10 monclient: get_auth_request con 0x7ff3ec06ce60 auth_method 0 2024-04-04T21:02:31.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.601+0000 7ff40affd640 1 --2- 172.21.15.106:0/239103095 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff3ec06ce60 0x7ff3ec06f320 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc001560 tx=0x7ff3fc007990 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.601+0000 7ff4037fe640 10 client.4349.objecter ms_handle_connect 0x7ff3ec06ce60 2024-04-04T21:02:31.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.603+0000 7ff4037fe640 1 -- 172.21.15.106:0/239103095 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff404056c20 con 0x7ff40c1d3a70 2024-04-04T21:02:31.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.603+0000 7ff4037fe640 10 client.4349.objecter ms_dispatch 0x7ff40c000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:31.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.603+0000 7ff4037fe640 3 client.4349.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:31.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.603+0000 7ff4037fe640 3 client.4349.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:31.604 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.603+0000 7ff4037fe640 20 client.4349.objecter dump_active .. 0 homeless 2024-04-04T21:02:31.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.610+0000 7f658e7fc640 1 -- 172.21.15.106:0/2375176186 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6594058ac0 con 0x7f6584004e10 2024-04-04T21:02:31.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.610+0000 7f658e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.610+0000 7f658e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:31.625 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.624+0000 7fb30d452640 1 --2- 172.21.15.106:0/2123984874 >> v2:172.21.15.203:3300/0 conn(0x7fb3080b0c80 0x7fb3080725f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.655 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.654+0000 7f657e36a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-04T21:02:31.655 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.654+0000 7f657e36a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-04T21:02:31.655 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.654+0000 7f657e36a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.655 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.654+0000 7f657e36a640 1 -- 172.21.15.106:0/2383310870 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f6578143520 con 0x7f65780b0b10 2024-04-04T21:02:31.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.677+0000 7fb30d452640 10 monclient(hunting): get_auth_request con 0x7fb3080b0c80 auth_method 0 2024-04-04T21:02:31.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.677+0000 7fb30d452640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.677+0000 7fb30d452640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.677+0000 7fb30d452640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.678 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-04T21:02:31.679 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.679+0000 7fb30d452640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.679 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.679+0000 7fb30d452640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.679 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.679+0000 7fb30d452640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.681+0000 7fb30cc51640 1 -- 172.21.15.106:0/2123984874 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476409116 0 0) 0x7fb3080b14b0 con 0x7fb308070fc0 2024-04-04T21:02:31.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.681+0000 7fb30cc51640 10 monclient(hunting): _negotiate 2024-04-04T21:02:31.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.681+0000 7fb30cc51640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.681+0000 7fb30cc51640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.681+0000 7fb30cc51640 10 monclient(hunting): my global_id is 4411 2024-04-04T21:02:31.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.681+0000 7fb30cc51640 1 -- 172.21.15.106:0/2123984874 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2f4003520 con 0x7fb308070fc0 2024-04-04T21:02:31.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.683+0000 7fb30d452640 10 monclient(hunting): handle_auth_done global_id 4408 payload 274 2024-04-04T21:02:31.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.683+0000 7fb30d452640 1 -- 172.21.15.106:0/2123984874 >> v1:172.21.15.203:6789/0 conn(0x7fb308070fc0 legacy=0x7fb3080aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.683+0000 7fb30d452640 1 -- 172.21.15.106:0/2123984874 >> v1:172.21.15.106:6789/0 conn(0x7fb3080b08a0 legacy=0x7fb30813fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.683+0000 7fb30d452640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.683+0000 7fb30d452640 1 monclient: found mon.noname-c 2024-04-04T21:02:31.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.683+0000 7fb30d452640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.683+0000 7fb30d452640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:02:31.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.683+0000 7fb30d452640 1 -- 172.21.15.106:0/2123984874 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30813f4e0 con 0x7fb3080b0c80 2024-04-04T21:02:31.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.689+0000 7fb30d452640 1 --2- 172.21.15.106:0/2123984874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080b0c80 0x7fb3080725f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80097a0 tx=0x7fb2f802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d92c22d4dc987fd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30cc51640 1 -- 172.21.15.106:0/2123984874 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8035020 con 0x7fb3080b0c80 2024-04-04T21:02:31.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30cc51640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30cc51640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:31.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30cc51640 10 monclient: dump: 2024-04-04T21:02:31.694 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.695 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.695 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.695 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.695 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.695 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.695 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.695 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.695 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.695 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30cc51640 10 monclient: _finish_auth 0 2024-04-04T21:02:31.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30cc51640 10 monclient: _check_auth_tickets 2024-04-04T21:02:31.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30cc51640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:31.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30cc51640 1 -- 172.21.15.106:0/2123984874 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f8031830 con 0x7fb3080b0c80 2024-04-04T21:02:31.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30feb7640 5 monclient: authenticate success, global_id 4408 2024-04-04T21:02:31.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30feb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:31.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30cc51640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:31.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30cc51640 1 -- 172.21.15.106:0/2123984874 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8031c00 con 0x7fb3080b0c80 2024-04-04T21:02:31.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30cc51640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30cc51640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:31.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30cc51640 10 monclient: dump: 2024-04-04T21:02:31.696 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.697 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.697 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.697 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.697 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.697 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.697 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.698 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.698 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.698 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.698 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30feb7640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:31.698 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.693+0000 7fb30feb7640 4 set_mon_vals no callback set 2024-04-04T21:02:31.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.694+0000 7fb30feb7640 10 monclient: shutdown 2024-04-04T21:02:31.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.694+0000 7fb30feb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:31.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.694+0000 7fb30feb7640 1 -- 172.21.15.106:0/2123984874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080b0c80 msgr2=0x7fb3080725f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.694+0000 7fb30feb7640 1 --2- 172.21.15.106:0/2123984874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080b0c80 0x7fb3080725f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80097a0 tx=0x7fb2f802b8d0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.694+0000 7fb30feb7640 1 -- 172.21.15.106:0/2123984874 shutdown_connections 2024-04-04T21:02:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.694+0000 7fb30feb7640 1 --2- 172.21.15.106:0/2123984874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080b0c80 0x7fb3080725f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.694+0000 7fb30feb7640 1 -- 172.21.15.106:0/2123984874 >> 172.21.15.106:0/2123984874 conn(0x7fb3080b1990 msgr2=0x7fb3080b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.695+0000 7fb30feb7640 1 -- 172.21.15.106:0/2123984874 shutdown_connections 2024-04-04T21:02:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.695+0000 7fb30feb7640 1 -- 172.21.15.106:0/2123984874 wait complete. 2024-04-04T21:02:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.695+0000 7fb30feb7640 10 monclient: build_initial_monmap 2024-04-04T21:02:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.695+0000 7fb30feb7640 10 monclient: monmap: 2024-04-04T21:02:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.703 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:31.703 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:31.703 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:31.703 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.696+0000 7fb30feb7640 1 librados: starting msgr at 2024-04-04T21:02:31.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.696+0000 7fb30feb7640 1 librados: starting objecter 2024-04-04T21:02:31.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.696+0000 7fb30feb7640 1 Processor -- start 2024-04-04T21:02:31.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.696+0000 7fb30feb7640 1 -- start start 2024-04-04T21:02:31.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.696+0000 7fb30feb7640 1 librados: setting wanted keys 2024-04-04T21:02:31.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.696+0000 7fb30feb7640 1 librados: calling monclient init 2024-04-04T21:02:31.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.696+0000 7fb30feb7640 10 monclient: init 2024-04-04T21:02:31.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.703 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-04T21:02:31.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb308070fc0 0x7fb308136ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 10 monclient(hunting): picked mon.noname-a con 0x7fb308070fc0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:31.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080b08a0 0x7fb30806af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 10 monclient(hunting): picked mon.noname-b con 0x7fb3080b08a0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:31.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3080abb80 0x7fb3080aad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 10 monclient(hunting): picked mon.noname-c con 0x7fb3080abb80 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:31.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30814ad20 con 0x7fb308070fc0 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080af5a0 con 0x7fb3080b08a0 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb30814aea0 con 0x7fb3080abb80 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.697+0000 7fb30feb7640 10 monclient(hunting): authenticate will time out at 1400.964898s 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.698+0000 7fb30d452640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080b08a0 0x7fb30806af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.698+0000 7fb30d452640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080b08a0 0x7fb30806af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50546/0 (socket says 172.21.15.106:50546) 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.698+0000 7fb30d452640 1 -- 172.21.15.106:0/359645547 learned_addr learned my addr 172.21.15.106:0/359645547 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.698+0000 7fb30d452640 10 monclient(hunting): get_auth_request con 0x7fb3080b08a0 auth_method 0 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.698+0000 7fb30d452640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.698+0000 7fb30d452640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.698+0000 7fb30d452640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.698+0000 7fb30d452640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.698+0000 7fb30d452640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.698+0000 7fb30d452640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.702+0000 7fb30d452640 10 monclient(hunting): handle_auth_done global_id 4414 payload 824 2024-04-04T21:02:31.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.702+0000 7fb30d452640 1 -- 172.21.15.106:0/359645547 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3080abb80 msgr2=0x7fb3080aad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.702+0000 7fb30d452640 1 --2- 172.21.15.106:0/359645547 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3080abb80 0x7fb3080aad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.702+0000 7fb30d452640 1 -- 172.21.15.106:0/359645547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb308070fc0 msgr2=0x7fb308136ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.702+0000 7fb30d452640 1 --2- 172.21.15.106:0/359645547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb308070fc0 0x7fb308136ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.702+0000 7fb30d452640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.702+0000 7fb30d452640 1 monclient: found mon.noname-b 2024-04-04T21:02:31.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.702+0000 7fb30d452640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.702+0000 7fb30d452640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:02:31.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.702+0000 7fb30d452640 1 -- 172.21.15.106:0/359645547 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3080aa850 con 0x7fb3080b08a0 2024-04-04T21:02:31.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.705+0000 7fb30d452640 1 --2- 172.21.15.106:0/359645547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080b08a0 0x7fb30806af80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8034040 tx=0x7fb2f80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.705+0000 7fb2fe7fc640 10 client.?.objecter ms_handle_connect 0x7fb3080b08a0 2024-04-04T21:02:31.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.705+0000 7fb2fe7fc640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:31.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb2fe7fc640 1 -- 172.21.15.106:0/359645547 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8040070 con 0x7fb3080b08a0 2024-04-04T21:02:31.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb2fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb2fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:31.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb2fe7fc640 10 monclient: dump: 2024-04-04T21:02:31.709 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.709 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.709 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.709 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.709 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.709 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.710 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb2fe7fc640 10 monclient: _finish_auth 0 2024-04-04T21:02:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb2fe7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:02:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb2fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb30feb7640 5 monclient: authenticate success, global_id 4414 2024-04-04T21:02:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb30feb7640 10 monclient: _renew_subs 2024-04-04T21:02:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb30feb7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb30feb7640 1 -- 172.21.15.106:0/359645547 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb30806bb70 con 0x7fb3080b08a0 2024-04-04T21:02:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb30feb7640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb30feb7640 10 monclient: _renew_subs 2024-04-04T21:02:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb30feb7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.707+0000 7fb30feb7640 1 -- 172.21.15.106:0/359645547 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3080aa470 con 0x7fb3080b08a0 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.708+0000 7fb30feb7640 1 librados: init done 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.708+0000 7fb2fe7fc640 1 -- 172.21.15.106:0/359645547 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f8031830 con 0x7fb3080b08a0 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.708+0000 7fb2fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.708+0000 7fb2fffff640 4 set_mon_vals no callback set 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.708+0000 7fb2fe7fc640 1 -- 172.21.15.106:0/359645547 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8035040 con 0x7fb3080b08a0 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.708+0000 7fb2fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.708+0000 7fb2fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.708+0000 7fb2fe7fc640 10 monclient: dump: 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.711 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.712 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.712 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.712 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.712 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.709+0000 7fb30feb7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.709+0000 7fb30feb7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.709+0000 7fb30feb7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:31.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.709+0000 7fb30feb7640 1 -- 172.21.15.106:0/359645547 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f0005080 con 0x7fb3080b08a0 2024-04-04T21:02:31.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.727+0000 7fb2fe7fc640 1 -- 172.21.15.106:0/359645547 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb2f8038020 con 0x7fb3080b08a0 2024-04-04T21:02:31.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.727+0000 7fb2fe7fc640 1 --2- 172.21.15.106:0/359645547 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb2ec06ce60 0x7fb2ec06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.728+0000 7fb30dc53640 1 --2- 172.21.15.106:0/359645547 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb2ec06ce60 0x7fb2ec06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.728+0000 7fb30dc53640 10 monclient: get_auth_request con 0x7fb2ec06ce60 auth_method 0 2024-04-04T21:02:31.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.728+0000 7fb2fe7fc640 10 client.4414.objecter ms_handle_connect 0x7fb2ec06ce60 2024-04-04T21:02:31.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.728+0000 7fb30dc53640 1 --2- 172.21.15.106:0/359645547 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb2ec06ce60 0x7fb2ec06f320 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb3040015f0 tx=0x7fb304007a40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.729+0000 7fb2fe7fc640 1 -- 172.21.15.106:0/359645547 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb2f8078d30 con 0x7fb3080b08a0 2024-04-04T21:02:31.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.729+0000 7fb2fe7fc640 10 client.4414.objecter ms_dispatch 0x7fb308000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:31.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.729+0000 7fb2fe7fc640 3 client.4414.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:31.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.729+0000 7fb2fe7fc640 3 client.4414.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:31.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.729+0000 7fb2fe7fc640 20 client.4414.objecter dump_active .. 0 homeless 2024-04-04T21:02:31.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.786+0000 7f9f6c206640 10 monclient: get_monmap_and_config 2024-04-04T21:02:31.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.786+0000 7f9f6c206640 10 monclient: build_initial_monmap 2024-04-04T21:02:31.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.786+0000 7f9f6c206640 10 monclient: monmap: 2024-04-04T21:02:31.789 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:31.789 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.789 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:31.787841+0000 2024-04-04T21:02:31.789 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:31.787841+0000 2024-04-04T21:02:31.789 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:31.789 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.790 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:31.790 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:31.790 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:31.790 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:31.790 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:31.791 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.787+0000 7f9f6c206640 1 Processor -- start 2024-04-04T21:02:31.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.787+0000 7f9f6c206640 1 -- start start 2024-04-04T21:02:31.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.787+0000 7f9f6c206640 10 monclient: init 2024-04-04T21:02:31.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.788+0000 7f9f6c206640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.788+0000 7f9f6c206640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-04-04T21:02:31.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.788+0000 7f9f6c206640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9f640a9b60 0x7f9f641430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.788+0000 7f9f6c206640 10 monclient(hunting): picked mon.noname-c con 0x7f9f640a9b60 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:31.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.788+0000 7f9f6c206640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f641426f0 0x7f9f640b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.788+0000 7f9f6c206640 10 monclient(hunting): picked mon.noname-b con 0x7f9f641426f0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:31.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.788+0000 7f9f6c206640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9f6413f810 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:31.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.788+0000 7f9f6c206640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f641435e0 con 0x7f9f6413f810 2024-04-04T21:02:31.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.788+0000 7f9f6c206640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.788+0000 7f9f6c206640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f640a9f40 con 0x7f9f640a9b60 2024-04-04T21:02:31.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.788+0000 7f9f6c206640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.788+0000 7f9f6c206640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f64142c10 con 0x7f9f641426f0 2024-04-04T21:02:31.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.788+0000 7f9f6c206640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.788+0000 7f9f6c206640 10 monclient(hunting): authenticate will time out at 1401.055345s 2024-04-04T21:02:31.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.813+0000 7f659fdfb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-04T21:02:31.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.813+0000 7f659fdfb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-04T21:02:31.814 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.813+0000 7f659fdfb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:31.815 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.813+0000 7f659fdfb640 1 -- 172.21.15.106:0/2375176186 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f659804f050 con 0x7f6584004e10 2024-04-04T21:02:31.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.855+0000 7ff4037fe640 1 -- 172.21.15.106:0/239103095 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff40c0b5cc0 con 0x7ff40c1d3a70 2024-04-04T21:02:31.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.855+0000 7ff4037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.856 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.855+0000 7ff4037fe640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:31.859 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-04T21:02:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.864+0000 7f6574ff9640 1 -- 172.21.15.106:0/2383310870 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f656805c360 con 0x7f65780b0b10 2024-04-04T21:02:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.865+0000 7f6574ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-04T21:02:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.865+0000 7f6574ff9640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.864+0000 7f9f697a1640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f641426f0 0x7f9f640b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.866 INFO:teuthology.orchestra.run.smithi106.stdout:64424509447 2024-04-04T21:02:31.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.865+0000 7f9f697a1640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f641426f0 0x7f9f640b28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:33176/0 (socket says 172.21.15.106:33176) 2024-04-04T21:02:31.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.865+0000 7f9f697a1640 1 -- 172.21.15.106:0/2437277583 learned_addr learned my addr 172.21.15.106:0/2437277583 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:31.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.865+0000 7f9f697a1640 10 monclient(hunting): get_auth_request con 0x7f9f641426f0 auth_method 0 2024-04-04T21:02:31.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.865+0000 7f9f697a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.865+0000 7f9f697a1640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.865+0000 7f9f697a1640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.866+0000 7f9f697a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.866+0000 7f9f697a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.867 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.866+0000 7f9f697a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 10 librados: watch_flush enter 2024-04-04T21:02:31.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 10 librados: watch_flush exit 2024-04-04T21:02:31.868 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 20 client.4337.objecter shutdown clearing up homeless session... 2024-04-04T21:02:31.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 10 client.4337.objecter successfully canceled tick 2024-04-04T21:02:31.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 1 -- 172.21.15.106:0/2383310870 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f656406ce60 msgr2=0x7f656406f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 1 --2- 172.21.15.106:0/2383310870 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f656406ce60 0x7f656406f320 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f656c0011e0 tx=0x7f656c009040 comp rx=0 tx=0).stop 2024-04-04T21:02:31.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 10 monclient: shutdown 2024-04-04T21:02:31.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:31.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 1 -- 172.21.15.106:0/2383310870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65780b0b10 msgr2=0x7f6578142b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 1 --2- 172.21.15.106:0/2383310870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65780b0b10 0x7f6578142b80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f656800c3f0 tx=0x7f656800c8b0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 1 -- 172.21.15.106:0/2383310870 shutdown_connections 2024-04-04T21:02:31.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 1 --2- 172.21.15.106:0/2383310870 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f656406ce60 0x7f656406f320 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 1 --2- 172.21.15.106:0/2383310870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65780b14b0 0x7f6578136f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 1 --2- 172.21.15.106:0/2383310870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65780b0b10 0x7f6578142b80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 1 --2- 172.21.15.106:0/2383310870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65780ae460 0x7f6578145650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 1 -- 172.21.15.106:0/2383310870 >> 172.21.15.106:0/2383310870 conn(0x7f65780b1990 msgr2=0x7f657813f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:31.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 1 -- 172.21.15.106:0/2383310870 shutdown_connections 2024-04-04T21:02:31.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 1 -- 172.21.15.106:0/2383310870 wait complete. 2024-04-04T21:02:31.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.867+0000 7f657e36a640 1 librados: shutdown 2024-04-04T21:02:31.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.869+0000 7f9f697a1640 10 monclient(hunting): handle_auth_done global_id 4352 payload 274 2024-04-04T21:02:31.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.869+0000 7f9f697a1640 1 -- 172.21.15.106:0/2437277583 >> v2:172.21.15.203:3300/0 conn(0x7f9f640a9b60 msgr2=0x7f9f641430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:31.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.869+0000 7f9f697a1640 1 --2- 172.21.15.106:0/2437277583 >> v2:172.21.15.203:3300/0 conn(0x7f9f640a9b60 0x7f9f641430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.869+0000 7f9f697a1640 1 -- 172.21.15.106:0/2437277583 >> v1:172.21.15.203:6789/0 conn(0x7f9f6413f810 legacy=0x7f9f6413fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:31.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.869+0000 7f9f697a1640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.869+0000 7f9f697a1640 1 monclient: found mon.noname-b 2024-04-04T21:02:31.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.869+0000 7f9f697a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.869+0000 7f9f697a1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.871 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.869+0000 7f9f697a1640 1 -- 172.21.15.106:0/2437277583 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f640b2e00 con 0x7f9f641426f0 2024-04-04T21:02:31.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.873+0000 7f9f697a1640 1 --2- 172.21.15.106:0/2437277583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f641426f0 0x7f9f640b28c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009870 tx=0x7f9f5402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11c209c4ed16f5d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.881 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.0 2024-04-04T21:02:31.882 DEBUG:teuthology.parallel:result is None 2024-04-04T21:02:31.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f68fa0640 1 -- 172.21.15.106:0/2437277583 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f540317c0 con 0x7f9f641426f0 2024-04-04T21:02:31.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f68fa0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f68fa0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:31.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f68fa0640 10 monclient: dump: 2024-04-04T21:02:31.884 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.885 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.885 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.885 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.885 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.885 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.885 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.885 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.885 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.885 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f68fa0640 10 monclient: _finish_auth 0 2024-04-04T21:02:31.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f68fa0640 10 monclient: _check_auth_tickets 2024-04-04T21:02:31.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f68fa0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:31.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f68fa0640 1 -- 172.21.15.106:0/2437277583 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f54037070 con 0x7f9f641426f0 2024-04-04T21:02:31.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f68fa0640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:31.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f68fa0640 1 -- 172.21.15.106:0/2437277583 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f54031b80 con 0x7f9f641426f0 2024-04-04T21:02:31.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f6c206640 5 monclient: authenticate success, global_id 4352 2024-04-04T21:02:31.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f6c206640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:31.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f6c206640 4 set_mon_vals no callback set 2024-04-04T21:02:31.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f68fa0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f68fa0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:31.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f68fa0640 10 monclient: dump: 2024-04-04T21:02:31.886 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.886 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.886 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.886 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.886 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.886 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.887 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.887 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.887 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.887 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f6c206640 10 monclient: shutdown 2024-04-04T21:02:31.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f6c206640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:31.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f6c206640 1 -- 172.21.15.106:0/2437277583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f641426f0 msgr2=0x7f9f640b28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f6c206640 1 --2- 172.21.15.106:0/2437277583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f641426f0 0x7f9f640b28c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009870 tx=0x7f9f5402b860 comp rx=0 tx=0).stop 2024-04-04T21:02:31.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f6c206640 1 -- 172.21.15.106:0/2437277583 shutdown_connections 2024-04-04T21:02:31.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f6c206640 1 --2- 172.21.15.106:0/2437277583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f641426f0 0x7f9f640b28c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f6c206640 1 --2- 172.21.15.106:0/2437277583 >> v2:172.21.15.203:3300/0 conn(0x7f9f640a9b60 0x7f9f641430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.881+0000 7f9f6c206640 1 -- 172.21.15.106:0/2437277583 >> 172.21.15.106:0/2437277583 conn(0x7f9f640b7ec0 msgr2=0x7f9f640b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:31.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.882+0000 7f9f6c206640 1 -- 172.21.15.106:0/2437277583 shutdown_connections 2024-04-04T21:02:31.887 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.882+0000 7f9f6c206640 1 -- 172.21.15.106:0/2437277583 wait complete. 2024-04-04T21:02:31.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.883+0000 7f9f6c206640 10 monclient: build_initial_monmap 2024-04-04T21:02:31.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.883+0000 7f9f6c206640 10 monclient: monmap: 2024-04-04T21:02:31.888 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:31.888 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:31.888 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:31.888 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:31.888 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:31.888 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.888 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:31.888 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:31.888 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:31.888 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.883+0000 7f9f6c206640 1 librados: starting msgr at 2024-04-04T21:02:31.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.883+0000 7f9f6c206640 1 librados: starting objecter 2024-04-04T21:02:31.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.883+0000 7f9f6c206640 1 Processor -- start 2024-04-04T21:02:31.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.883+0000 7f9f6c206640 1 -- start start 2024-04-04T21:02:31.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.883+0000 7f9f6c206640 1 librados: setting wanted keys 2024-04-04T21:02:31.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.883+0000 7f9f6c206640 1 librados: calling monclient init 2024-04-04T21:02:31.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.883+0000 7f9f6c206640 10 monclient: init 2024-04-04T21:02:31.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-04T21:02:31.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f640a9b60 0x7f9f64139580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 10 monclient(hunting): picked mon.noname-a con 0x7f9f640a9b60 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:31.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f6413f810 0x7f9f64139ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 10 monclient(hunting): picked mon.noname-b con 0x7f9f6413f810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:31.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f64136130 0x7f9f641365a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 10 monclient(hunting): picked mon.noname-c con 0x7f9f64136130 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:31.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f640b2530 con 0x7f9f640a9b60 2024-04-04T21:02:31.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f5bc1ffb640 1 -- 172.21.15.106:0/2769814709 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5bc804c060 con 0x7f5bcc1442f0 2024-04-04T21:02:31.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f640b23b0 con 0x7f9f6413f810 2024-04-04T21:02:31.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6414d000 con 0x7f9f64136130 2024-04-04T21:02:31.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.884+0000 7f9f6c206640 10 monclient(hunting): authenticate will time out at 1401.152163s 2024-04-04T21:02:31.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.885+0000 7f5bc1ffb640 1 --2- 172.21.15.106:0/2769814709 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5bb006ce90 0x7f5bb006f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.885+0000 7f5bd0b60640 1 --2- 172.21.15.106:0/2769814709 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5bb006ce90 0x7f5bb006f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.885+0000 7f5bd0b60640 10 monclient: get_auth_request con 0x7f5bb006ce90 auth_method 0 2024-04-04T21:02:31.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.885+0000 7f5bd0b60640 1 --2- 172.21.15.106:0/2769814709 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5bb006ce90 0x7f5bb006f350 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc144050 tx=0x7f5bbc006ba0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.885+0000 7f5bc1ffb640 10 client.4365.objecter ms_handle_connect 0x7f5bb006ce90 2024-04-04T21:02:31.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.886+0000 7f9f6a7a3640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f64136130 0x7f9f641365a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.887+0000 7f9f6a7a3640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f64136130 0x7f9f641365a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:33184/0 (socket says 172.21.15.106:33184) 2024-04-04T21:02:31.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.887+0000 7f9f6a7a3640 1 -- 172.21.15.106:0/246167929 learned_addr learned my addr 172.21.15.106:0/246167929 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:31.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.887+0000 7f9f6a7a3640 10 monclient(hunting): get_auth_request con 0x7f9f64136130 auth_method 0 2024-04-04T21:02:31.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.888+0000 7f9f6a7a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.888+0000 7f9f6a7a3640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.888+0000 7f9f6a7a3640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.890+0000 7f9f6a7a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.890+0000 7f9f6a7a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.890+0000 7f9f6a7a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.894+0000 7f9f6a7a3640 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2024-04-04T21:02:31.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.894+0000 7f9f6a7a3640 1 -- 172.21.15.106:0/246167929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f6413f810 msgr2=0x7f9f64139ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.894+0000 7f9f6a7a3640 1 --2- 172.21.15.106:0/246167929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f6413f810 0x7f9f64139ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.894+0000 7f9f6a7a3640 1 -- 172.21.15.106:0/246167929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f640a9b60 msgr2=0x7f9f64139580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.894+0000 7f9f6a7a3640 1 --2- 172.21.15.106:0/246167929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f640a9b60 0x7f9f64139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.894+0000 7f9f6a7a3640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.894+0000 7f9f6a7a3640 1 monclient: found mon.noname-c 2024-04-04T21:02:31.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.894+0000 7f9f6a7a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.894+0000 7f9f6a7a3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.894+0000 7f9f6a7a3640 1 -- 172.21.15.106:0/246167929 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f64137a60 con 0x7f9f64136130 2024-04-04T21:02:31.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.896+0000 7f9f6a7a3640 1 --2- 172.21.15.106:0/246167929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f64136130 0x7f9f641365a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c00c780 tx=0x7f9f5c00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.896+0000 7f9f5a7fc640 10 client.?.objecter ms_handle_connect 0x7f9f64136130 2024-04-04T21:02:31.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.896+0000 7f9f5a7fc640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:31.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.901+0000 7f5bc1ffb640 1 -- 172.21.15.106:0/2769814709 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5bc8078280 con 0x7f5bcc1442f0 2024-04-04T21:02:31.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.901+0000 7f5bc1ffb640 10 client.4365.objecter ms_dispatch 0x7f5bcc000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:31.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.901+0000 7f5bc1ffb640 3 client.4365.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:31.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.901+0000 7f5bc1ffb640 3 client.4365.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:31.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.901+0000 7f5bc1ffb640 20 client.4365.objecter dump_active .. 0 homeless 2024-04-04T21:02:31.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.901+0000 7f5bc1ffb640 1 -- 172.21.15.106:0/2769814709 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5bcc141ab0 con 0x7f5bcc1442f0 2024-04-04T21:02:31.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.901+0000 7f5bc1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.901+0000 7f5bc1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:31.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f5a7fc640 1 -- 172.21.15.106:0/246167929 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c017070 con 0x7f9f64136130 2024-04-04T21:02:31.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f5a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:02:31.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f5a7fc640 10 monclient: dump: 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f5a7fc640 10 monclient: _finish_auth 0 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f5a7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f5a7fc640 1 -- 172.21.15.106:0/246167929 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5c003da0 con 0x7f9f64136130 2024-04-04T21:02:31.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:31.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f5a7fc640 1 -- 172.21.15.106:0/246167929 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c0052b0 con 0x7f9f64136130 2024-04-04T21:02:31.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f5a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:31.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f5a7fc640 10 monclient: dump: 2024-04-04T21:02:31.906 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.906 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.906 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.906 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.906 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.906 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.906 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.906 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.906 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.906 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f5bfff640 4 set_mon_vals no callback set 2024-04-04T21:02:31.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f6c206640 5 monclient: authenticate success, global_id 4355 2024-04-04T21:02:31.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f6c206640 10 monclient: _renew_subs 2024-04-04T21:02:31.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.902+0000 7f9f6c206640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.903+0000 7f9f6c206640 1 -- 172.21.15.106:0/246167929 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f64137d40 con 0x7f9f64136130 2024-04-04T21:02:31.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.903+0000 7f9f6c206640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:31.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.903+0000 7f9f6c206640 10 monclient: _renew_subs 2024-04-04T21:02:31.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.903+0000 7f9f6c206640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.903+0000 7f9f6c206640 1 -- 172.21.15.106:0/246167929 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f64132130 con 0x7f9f64136130 2024-04-04T21:02:31.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.903+0000 7f9f6c206640 1 librados: init done 2024-04-04T21:02:31.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.904+0000 7f9f6c206640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.904+0000 7f9f6c206640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.904+0000 7f9f6c206640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:31.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.904+0000 7f9f6c206640 1 -- 172.21.15.106:0/246167929 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f64135cc0 con 0x7f9f64136130 2024-04-04T21:02:31.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.927+0000 7f9f5a7fc640 1 -- 172.21.15.106:0/246167929 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9f5c01a070 con 0x7f9f64136130 2024-04-04T21:02:31.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.928+0000 7f9f5a7fc640 1 --2- 172.21.15.106:0/246167929 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f9f4806ceb0 0x7f9f4806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.928+0000 7f9f69fa2640 1 --2- 172.21.15.106:0/246167929 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f9f4806ceb0 0x7f9f4806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.928+0000 7f9f69fa2640 10 monclient: get_auth_request con 0x7f9f4806ceb0 auth_method 0 2024-04-04T21:02:31.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.929+0000 7f9f5a7fc640 10 client.4355.objecter ms_handle_connect 0x7f9f4806ceb0 2024-04-04T21:02:31.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.929+0000 7f9f69fa2640 1 --2- 172.21.15.106:0/246167929 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f9f4806ceb0 0x7f9f4806f370 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009710 tx=0x7f9f60006c90 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.931+0000 7f9f5a7fc640 1 -- 172.21.15.106:0/246167929 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9f5c0564e0 con 0x7f9f64136130 2024-04-04T21:02:31.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.931+0000 7f9f5a7fc640 10 client.4355.objecter ms_dispatch 0x7f9f64000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:31.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.931+0000 7f9f5a7fc640 3 client.4355.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:31.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.931+0000 7f9f5a7fc640 3 client.4355.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:31.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.931+0000 7f9f5a7fc640 20 client.4355.objecter dump_active .. 0 homeless 2024-04-04T21:02:31.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.969+0000 7fdfa5e49640 10 monclient: get_monmap_and_config 2024-04-04T21:02:31.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.969+0000 7fdfa5e49640 10 monclient: build_initial_monmap 2024-04-04T21:02:31.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.969+0000 7fdfa5e49640 10 monclient: monmap: 2024-04-04T21:02:31.970 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:31.970 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.970 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:31.970977+0000 2024-04-04T21:02:31.971 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:31.970977+0000 2024-04-04T21:02:31.971 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:31.971 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.971 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:31.971 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:31.971 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:31.971 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:31.971 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:31.971 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.970+0000 7fdfa5e49640 1 Processor -- start 2024-04-04T21:02:31.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.970+0000 7fdfa5e49640 1 -- start start 2024-04-04T21:02:31.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.970+0000 7fdfa5e49640 10 monclient: init 2024-04-04T21:02:31.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.971+0000 7fdfa5e49640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.971+0000 7fdfa5e49640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-04-04T21:02:31.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.971+0000 7fdfa5e49640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdfa00a9b60 0x7fdfa01430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.971+0000 7fdfa5e49640 10 monclient(hunting): picked mon.noname-a con 0x7fdfa00a9b60 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:31.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.971+0000 7fdfa5e49640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdfa01426f0 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:31.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.971+0000 7fdfa5e49640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdfa013f810 0x7fdfa013fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.971+0000 7fdfa5e49640 10 monclient(hunting): picked mon.noname-b con 0x7fdfa013f810 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:31.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.971+0000 7fdfa5e49640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa01435e0 con 0x7fdfa01426f0 2024-04-04T21:02:31.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.971+0000 7fdfa5e49640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.971+0000 7fdfa5e49640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa00a9f40 con 0x7fdfa00a9b60 2024-04-04T21:02:31.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.971+0000 7fdfa5e49640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.971+0000 7fdfa5e49640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa0142c10 con 0x7fdfa013f810 2024-04-04T21:02:31.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.971+0000 7fdfa5e49640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.971+0000 7fdfa5e49640 10 monclient(hunting): authenticate will time out at 1401.238533s 2024-04-04T21:02:31.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.972+0000 7fdf9f7fe640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdfa00a9b60 0x7fdfa01430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.972+0000 7fdf9f7fe640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdfa00a9b60 0x7fdfa01430a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35152/0 (socket says 172.21.15.106:35152) 2024-04-04T21:02:31.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.972+0000 7fdf9f7fe640 1 -- 172.21.15.106:0/2334057630 learned_addr learned my addr 172.21.15.106:0/2334057630 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:31.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.972+0000 7fdf9f7fe640 10 monclient(hunting): get_auth_request con 0x7fdfa00a9b60 auth_method 0 2024-04-04T21:02:31.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.972+0000 7fdf9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.972+0000 7fdf9f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.972+0000 7fdf9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.975 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.972+0000 7fdf9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.975 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.972+0000 7fdf9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.975 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.972+0000 7fdf9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.974+0000 7fdf9f7fe640 10 monclient(hunting): handle_auth_done global_id 4371 payload 274 2024-04-04T21:02:31.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.974+0000 7fdf9f7fe640 1 -- 172.21.15.106:0/2334057630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdfa013f810 msgr2=0x7fdfa013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:31.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.974+0000 7fdf9f7fe640 1 --2- 172.21.15.106:0/2334057630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdfa013f810 0x7fdfa013fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.974+0000 7fdf9f7fe640 1 -- 172.21.15.106:0/2334057630 >> v1:172.21.15.203:6789/0 conn(0x7fdfa01426f0 legacy=0x7fdfa00b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:31.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.974+0000 7fdf9f7fe640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.974+0000 7fdf9f7fe640 1 monclient: found mon.noname-a 2024-04-04T21:02:31.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.974+0000 7fdf9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.974+0000 7fdf9f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:31.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.974+0000 7fdf9f7fe640 1 -- 172.21.15.106:0/2334057630 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa00b8c90 con 0x7fdfa00a9b60 2024-04-04T21:02:31.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.976+0000 7fdf9f7fe640 1 --2- 172.21.15.106:0/2334057630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfa00a9b60 0x7fdfa01430a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009870 tx=0x7fdf9002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f5ac47ae307652 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdf9e7fc640 1 -- 172.21.15.106:0/2334057630 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf900317c0 con 0x7fdfa00a9b60 2024-04-04T21:02:31.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdf9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdf9e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:31.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdf9e7fc640 10 monclient: dump: 2024-04-04T21:02:31.981 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.981 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.981 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.982 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.982 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.982 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.982 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.982 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.982 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.982 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdf9e7fc640 10 monclient: _finish_auth 0 2024-04-04T21:02:31.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdf9e7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:02:31.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdf9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:31.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdf9e7fc640 1 -- 172.21.15.106:0/2334057630 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf90037070 con 0x7fdfa00a9b60 2024-04-04T21:02:31.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdfa5e49640 5 monclient: authenticate success, global_id 4371 2024-04-04T21:02:31.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdfa5e49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:02:31.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdf9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:31.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdfa5e49640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:31.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdf9e7fc640 1 -- 172.21.15.106:0/2334057630 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90031b80 con 0x7fdfa00a9b60 2024-04-04T21:02:31.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdf9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:31.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdfa5e49640 4 set_mon_vals no callback set 2024-04-04T21:02:31.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdf9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:31.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.980+0000 7fdf9e7fc640 10 monclient: dump: 2024-04-04T21:02:31.984 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:31.984 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:31.984 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.984 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:31.984 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:31.984 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.984 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:31.984 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:31.984 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:31.984 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.981+0000 7fdfa5e49640 10 monclient: shutdown 2024-04-04T21:02:31.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.981+0000 7fdfa5e49640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:31.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.981+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2334057630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfa00a9b60 msgr2=0x7fdfa01430a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.981+0000 7fdfa5e49640 1 --2- 172.21.15.106:0/2334057630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfa00a9b60 0x7fdfa01430a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009870 tx=0x7fdf9002b860 comp rx=0 tx=0).stop 2024-04-04T21:02:31.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.981+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2334057630 shutdown_connections 2024-04-04T21:02:31.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.981+0000 7fdfa5e49640 1 --2- 172.21.15.106:0/2334057630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdfa013f810 0x7fdfa013fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.981+0000 7fdfa5e49640 1 --2- 172.21.15.106:0/2334057630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfa00a9b60 0x7fdfa01430a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.981+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2334057630 >> 172.21.15.106:0/2334057630 conn(0x7fdfa00b7ec0 msgr2=0x7fdfa00b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:31.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.981+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2334057630 shutdown_connections 2024-04-04T21:02:31.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.981+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2334057630 wait complete. 2024-04-04T21:02:31.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.982+0000 7fdfa5e49640 10 monclient: build_initial_monmap 2024-04-04T21:02:31.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.982+0000 7fdfa5e49640 10 monclient: monmap: 2024-04-04T21:02:31.987 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:31.987 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:31.987 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:31.987 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:31.987 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:31.987 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:31.987 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:31.988 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:31.988 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:31.988 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:31.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.982+0000 7fdfa5e49640 1 librados: starting msgr at 2024-04-04T21:02:31.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.982+0000 7fdfa5e49640 1 librados: starting objecter 2024-04-04T21:02:31.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.982+0000 7fdfa5e49640 1 Processor -- start 2024-04-04T21:02:31.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 1 -- start start 2024-04-04T21:02:31.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 1 librados: setting wanted keys 2024-04-04T21:02:31.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 1 librados: calling monclient init 2024-04-04T21:02:31.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 10 monclient: init 2024-04-04T21:02:31.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fb2fe7fc640 1 -- 172.21.15.106:0/359645547 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb2f8052070 con 0x7fb3080b08a0 2024-04-04T21:02:31.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fb2fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:31.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fb2fe7fc640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:31.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfa00a9b60 0x7fdfa0136d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 10 monclient(hunting): picked mon.noname-a con 0x7fdfa00a9b60 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfa013f810 0x7fdfa013a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 10 monclient(hunting): picked mon.noname-b con 0x7fdfa013f810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdfa01372d0 0x7fdfa013a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 10 monclient(hunting): picked mon.noname-c con 0x7fdfa01372d0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00b2530 con 0x7fdfa00a9b60 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00b23b0 con 0x7fdfa013f810 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa014d000 con 0x7fdfa01372d0 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:31.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.983+0000 7fdfa5e49640 10 monclient(hunting): authenticate will time out at 1401.251029s 2024-04-04T21:02:31.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.984+0000 7fdf9f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfa00a9b60 0x7fdfa0136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:31.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.984+0000 7fdf9f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfa00a9b60 0x7fdfa0136d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35168/0 (socket says 172.21.15.106:35168) 2024-04-04T21:02:31.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.984+0000 7fdf9f7fe640 1 -- 172.21.15.106:0/2156704345 learned_addr learned my addr 172.21.15.106:0/2156704345 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:31.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.984+0000 7fdf9f7fe640 10 monclient(hunting): get_auth_request con 0x7fdfa00a9b60 auth_method 0 2024-04-04T21:02:31.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.984+0000 7fdf9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:31.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.984+0000 7fdf9f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:31.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.984+0000 7fdf9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:31.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.985+0000 7fdf9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:31.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.985+0000 7fdf9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:31.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.985+0000 7fdf9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:31.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.988+0000 7fdf9f7fe640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2024-04-04T21:02:31.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.988+0000 7fdf9f7fe640 1 -- 172.21.15.106:0/2156704345 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdfa01372d0 msgr2=0x7fdfa013a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.988+0000 7fdf9f7fe640 1 --2- 172.21.15.106:0/2156704345 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdfa01372d0 0x7fdfa013a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.988+0000 7fdf9f7fe640 1 -- 172.21.15.106:0/2156704345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfa013f810 msgr2=0x7fdfa013a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:31.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.988+0000 7fdf9f7fe640 1 --2- 172.21.15.106:0/2156704345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfa013f810 0x7fdfa013a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:31.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.988+0000 7fdf9f7fe640 10 monclient: _finish_hunting 0 2024-04-04T21:02:31.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.988+0000 7fdf9f7fe640 1 monclient: found mon.noname-a 2024-04-04T21:02:31.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.988+0000 7fdf9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:31.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.988+0000 7fdf9f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:31.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.988+0000 7fdf9f7fe640 1 -- 172.21.15.106:0/2156704345 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa0138040 con 0x7fdfa00a9b60 2024-04-04T21:02:31.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.996+0000 7fdf9f7fe640 1 --2- 172.21.15.106:0/2156704345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfa00a9b60 0x7fdfa0136d90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdf90000c00 tx=0x7fdf900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:31.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.996+0000 7fdf87fff640 10 client.?.objecter ms_handle_connect 0x7fdfa00a9b60 2024-04-04T21:02:31.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.996+0000 7fdf87fff640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:31.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.999+0000 7f658e7fc640 1 -- 172.21.15.106:0/2375176186 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f659405cb60 con 0x7f6584004e10 2024-04-04T21:02:31.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.999+0000 7f658e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-04T21:02:31.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:31.999+0000 7f658e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:31.999 INFO:teuthology.orchestra.run.smithi106.stdout:55834574854 2024-04-04T21:02:32.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.001+0000 7f659fdfb640 10 librados: watch_flush enter 2024-04-04T21:02:32.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 10 librados: watch_flush exit 2024-04-04T21:02:32.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 20 client.4405.objecter shutdown clearing up homeless session... 2024-04-04T21:02:32.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 10 client.4405.objecter successfully canceled tick 2024-04-04T21:02:32.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 1 -- 172.21.15.106:0/2375176186 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f657c06cee0 msgr2=0x7f657c06f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 1 --2- 172.21.15.106:0/2375176186 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f657c06cee0 0x7f657c06f3a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6588001eb0 tx=0x7f6588001600 comp rx=0 tx=0).stop 2024-04-04T21:02:32.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 10 monclient: shutdown 2024-04-04T21:02:32.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:32.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 1 -- 172.21.15.106:0/2375176186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6584004e10 msgr2=0x7f65980b6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 1 --2- 172.21.15.106:0/2375176186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6584004e10 0x7f65980b6d70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f659400c570 tx=0x7f659400ca30 comp rx=0 tx=0).stop 2024-04-04T21:02:32.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 1 -- 172.21.15.106:0/2375176186 shutdown_connections 2024-04-04T21:02:32.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 1 --2- 172.21.15.106:0/2375176186 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f657c06cee0 0x7f657c06f3a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 1 --2- 172.21.15.106:0/2375176186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65981db390 0x7f65981db810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 1 --2- 172.21.15.106:0/2375176186 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6598145530 0x7f6598145970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 1 --2- 172.21.15.106:0/2375176186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6584004e10 0x7f65980b6d70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 1 -- 172.21.15.106:0/2375176186 >> 172.21.15.106:0/2375176186 conn(0x7f65980b7ec0 msgr2=0x7f659813ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:32.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 1 -- 172.21.15.106:0/2375176186 shutdown_connections 2024-04-04T21:02:32.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.002+0000 7f659fdfb640 1 -- 172.21.15.106:0/2375176186 wait complete. 2024-04-04T21:02:32.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.003+0000 7f659fdfb640 1 librados: shutdown 2024-04-04T21:02:32.019 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-04-04T21:02:32.019 DEBUG:teuthology.parallel:result is None 2024-04-04T21:02:32.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdf87fff640 1 -- 172.21.15.106:0/2156704345 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90032860 con 0x7fdfa00a9b60 2024-04-04T21:02:32.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdf87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:32.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdf87fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:32.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdf87fff640 10 monclient: dump: 2024-04-04T21:02:32.028 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:32.028 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:32.028 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:32.029 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:32.029 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:32.029 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:32.029 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:32.029 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:32.029 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:32.029 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:32.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdf87fff640 10 monclient: _finish_auth 0 2024-04-04T21:02:32.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdf87fff640 10 monclient: _check_auth_tickets 2024-04-04T21:02:32.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdf87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:32.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdf87fff640 1 -- 172.21.15.106:0/2156704345 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf90032a20 con 0x7fdfa00a9b60 2024-04-04T21:02:32.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdf87fff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:32.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdf87fff640 1 -- 172.21.15.106:0/2156704345 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90032e10 con 0x7fdfa00a9b60 2024-04-04T21:02:32.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdf87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:32.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdf87fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:32.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdf87fff640 10 monclient: dump: 2024-04-04T21:02:32.034 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:32.034 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:32.034 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:32.034 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdfa5e49640 5 monclient: authenticate success, global_id 4374 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdfa5e49640 10 monclient: _renew_subs 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdfa5e49640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2156704345 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa0136210 con 0x7fdfa00a9b60 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdf9dffb640 4 set_mon_vals no callback set 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdfa5e49640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdfa5e49640 10 monclient: _renew_subs 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdfa5e49640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:32.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2156704345 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa013b040 con 0x7fdfa00a9b60 2024-04-04T21:02:32.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.027+0000 7fdfa5e49640 1 librados: init done 2024-04-04T21:02:32.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.028+0000 7fdfa5e49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:32.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.028+0000 7fdfa5e49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:32.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.028+0000 7fdfa5e49640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:32.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.028+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2156704345 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa004f050 con 0x7fdfa00a9b60 2024-04-04T21:02:32.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.140+0000 7ff4117a5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-04T21:02:32.141 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.140+0000 7ff4117a5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-04T21:02:32.141 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.140+0000 7ff4117a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:32.141 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.140+0000 7ff4117a5640 1 -- 172.21.15.106:0/239103095 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7ff40c1d3f00 con 0x7ff40c1d3a70 2024-04-04T21:02:32.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.171+0000 7fb30feb7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-04T21:02:32.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.171+0000 7fb30feb7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-04T21:02:32.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.171+0000 7fb30feb7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:02:32.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.171+0000 7fb30feb7640 1 -- 172.21.15.106:0/359645547 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb2f0005640 con 0x7fb3080b08a0 2024-04-04T21:02:32.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.175+0000 7fb2fe7fc640 1 -- 172.21.15.106:0/359645547 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb2f807b390 con 0x7fb3080b08a0 2024-04-04T21:02:32.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.175+0000 7fb2fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-04T21:02:32.175 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.175+0000 7fb2fe7fc640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:32.175 INFO:teuthology.orchestra.run.smithi106.stdout:85899345930 2024-04-04T21:02:32.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.176+0000 7fb30feb7640 10 librados: watch_flush enter 2024-04-04T21:02:32.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.176+0000 7fb30feb7640 10 librados: watch_flush exit 2024-04-04T21:02:32.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.176+0000 7fb30feb7640 20 client.4414.objecter shutdown clearing up homeless session... 2024-04-04T21:02:32.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.176+0000 7fb30feb7640 10 client.4414.objecter successfully canceled tick 2024-04-04T21:02:32.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 1 -- 172.21.15.106:0/359645547 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb2ec06ce60 msgr2=0x7fb2ec06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 1 --2- 172.21.15.106:0/359645547 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb2ec06ce60 0x7fb2ec06f320 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb3040015f0 tx=0x7fb304007a40 comp rx=0 tx=0).stop 2024-04-04T21:02:32.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 10 monclient: shutdown 2024-04-04T21:02:32.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:32.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 1 -- 172.21.15.106:0/359645547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080b08a0 msgr2=0x7fb30806af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 1 --2- 172.21.15.106:0/359645547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080b08a0 0x7fb30806af80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8034040 tx=0x7fb2f80046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 1 -- 172.21.15.106:0/359645547 shutdown_connections 2024-04-04T21:02:32.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 1 --2- 172.21.15.106:0/359645547 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb2ec06ce60 0x7fb2ec06f320 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 1 --2- 172.21.15.106:0/359645547 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3080abb80 0x7fb3080aad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 1 --2- 172.21.15.106:0/359645547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080b08a0 0x7fb30806af80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 1 --2- 172.21.15.106:0/359645547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb308070fc0 0x7fb308136ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 1 -- 172.21.15.106:0/359645547 >> 172.21.15.106:0/359645547 conn(0x7fb3080b1990 msgr2=0x7fb3080b3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:32.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 1 -- 172.21.15.106:0/359645547 shutdown_connections 2024-04-04T21:02:32.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 1 -- 172.21.15.106:0/359645547 wait complete. 2024-04-04T21:02:32.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.177+0000 7fb30feb7640 1 librados: shutdown 2024-04-04T21:02:32.188 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345930 for osd.1 2024-04-04T21:02:32.188 DEBUG:teuthology.parallel:result is None 2024-04-04T21:02:32.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.208+0000 7f5bd35c5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-04-04T21:02:32.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.208+0000 7f5bd35c5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-04-04T21:02:32.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.208+0000 7f5bd35c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:32.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.208+0000 7f5bd35c5640 1 -- 172.21.15.106:0/2769814709 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f5bcc06e370 con 0x7f5bcc1442f0 2024-04-04T21:02:32.222 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.221+0000 7f9f5a7fc640 1 -- 172.21.15.106:0/246167929 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9f64135cc0 con 0x7f9f64136130 2024-04-04T21:02:32.222 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.221+0000 7f9f5a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:32.222 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.221+0000 7f9f5a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:32.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.229+0000 7ff4037fe640 1 -- 172.21.15.106:0/239103095 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff404023020 con 0x7ff40c1d3a70 2024-04-04T21:02:32.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.229+0000 7ff4037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-04T21:02:32.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.229+0000 7ff4037fe640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:32.230 INFO:teuthology.orchestra.run.smithi106.stdout:60129542151 2024-04-04T21:02:32.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.231+0000 7ff4117a5640 10 librados: watch_flush enter 2024-04-04T21:02:32.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 10 librados: watch_flush exit 2024-04-04T21:02:32.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 20 client.4349.objecter shutdown clearing up homeless session... 2024-04-04T21:02:32.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 10 client.4349.objecter successfully canceled tick 2024-04-04T21:02:32.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 1 -- 172.21.15.106:0/239103095 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff3ec06ce60 msgr2=0x7ff3ec06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 1 --2- 172.21.15.106:0/239103095 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff3ec06ce60 0x7ff3ec06f320 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc001560 tx=0x7ff3fc007990 comp rx=0 tx=0).stop 2024-04-04T21:02:32.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 10 monclient: shutdown 2024-04-04T21:02:32.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:32.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 1 -- 172.21.15.106:0/239103095 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c1d3a70 msgr2=0x7ff40c140810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 1 --2- 172.21.15.106:0/239103095 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c1d3a70 0x7ff40c140810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff40400b970 tx=0x7ff40400be30 comp rx=0 tx=0).stop 2024-04-04T21:02:32.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 1 -- 172.21.15.106:0/239103095 shutdown_connections 2024-04-04T21:02:32.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 1 --2- 172.21.15.106:0/239103095 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff3ec06ce60 0x7ff3ec06f320 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 1 --2- 172.21.15.106:0/239103095 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c1d3a70 0x7ff40c140810 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 1 --2- 172.21.15.106:0/239103095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c143eb0 0x7ff40c1d3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 1 --2- 172.21.15.106:0/239103095 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff40c070e50 0x7ff40c13cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 1 -- 172.21.15.106:0/239103095 >> 172.21.15.106:0/239103095 conn(0x7ff40c0b62c0 msgr2=0x7ff40c06d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:32.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 1 -- 172.21.15.106:0/239103095 shutdown_connections 2024-04-04T21:02:32.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 1 -- 172.21.15.106:0/239103095 wait complete. 2024-04-04T21:02:32.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.232+0000 7ff4117a5640 1 librados: shutdown 2024-04-04T21:02:32.244 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.7 2024-04-04T21:02:32.245 DEBUG:teuthology.parallel:result is None 2024-04-04T21:02:32.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.406+0000 7fdf87fff640 1 -- 172.21.15.106:0/2156704345 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdf9004c060 con 0x7fdfa00a9b60 2024-04-04T21:02:32.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.406+0000 7fdf87fff640 1 --2- 172.21.15.106:0/2156704345 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdf8c06ce60 0x7fdf8c06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:32.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.406+0000 7fdf9effd640 1 --2- 172.21.15.106:0/2156704345 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdf8c06ce60 0x7fdf8c06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:32.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.406+0000 7fdf9effd640 10 monclient: get_auth_request con 0x7fdf8c06ce60 auth_method 0 2024-04-04T21:02:32.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.407+0000 7fdf9effd640 1 --2- 172.21.15.106:0/2156704345 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdf8c06ce60 0x7fdf8c06f320 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00b5bb0 tx=0x7fdf94006ba0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:32.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.407+0000 7fdf87fff640 10 client.4374.objecter ms_handle_connect 0x7fdf8c06ce60 2024-04-04T21:02:32.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.407+0000 7f9f6c206640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-04T21:02:32.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.407+0000 7f9f6c206640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-04T21:02:32.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.407+0000 7f9f6c206640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:02:32.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.407+0000 7f9f6c206640 1 -- 172.21.15.106:0/246167929 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f9f6404f050 con 0x7f9f64136130 2024-04-04T21:02:32.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.420+0000 7fdf87fff640 1 -- 172.21.15.106:0/2156704345 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdf90078740 con 0x7fdfa00a9b60 2024-04-04T21:02:32.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.420+0000 7fdf87fff640 10 client.4374.objecter ms_dispatch 0x7fdfa0000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:32.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.420+0000 7fdf87fff640 3 client.4374.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:32.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.420+0000 7fdf87fff640 3 client.4374.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:32.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.420+0000 7fdf87fff640 20 client.4374.objecter dump_active .. 0 homeless 2024-04-04T21:02:32.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.420+0000 7fdf87fff640 1 -- 172.21.15.106:0/2156704345 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdf90040070 con 0x7fdfa00a9b60 2024-04-04T21:02:32.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.420+0000 7fdf87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:32.422 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.420+0000 7fdf87fff640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:32.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.461+0000 7f9f5a7fc640 1 -- 172.21.15.106:0/246167929 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9f5c021020 con 0x7f9f64136130 2024-04-04T21:02:32.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.461+0000 7f9f5a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-04T21:02:32.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.461+0000 7f9f5a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:32.462 INFO:teuthology.orchestra.run.smithi106.stdout:85899345929 2024-04-04T21:02:32.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.463+0000 7f5bc1ffb640 1 -- 172.21.15.106:0/2769814709 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5bc8040070 con 0x7f5bcc1442f0 2024-04-04T21:02:32.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.463+0000 7f5bc1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-04-04T21:02:32.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.463+0000 7f5bc1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:32.463 INFO:teuthology.orchestra.run.smithi106.stdout:64424509449 2024-04-04T21:02:32.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 10 librados: watch_flush enter 2024-04-04T21:02:32.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 10 librados: watch_flush exit 2024-04-04T21:02:32.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 20 client.4355.objecter shutdown clearing up homeless session... 2024-04-04T21:02:32.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 10 client.4355.objecter successfully canceled tick 2024-04-04T21:02:32.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 1 -- 172.21.15.106:0/246167929 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f9f4806ceb0 msgr2=0x7f9f4806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 1 --2- 172.21.15.106:0/246167929 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f9f4806ceb0 0x7f9f4806f370 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009710 tx=0x7f9f60006c90 comp rx=0 tx=0).stop 2024-04-04T21:02:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 10 monclient: shutdown 2024-04-04T21:02:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 1 -- 172.21.15.106:0/246167929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f64136130 msgr2=0x7f9f641365a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 1 --2- 172.21.15.106:0/246167929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f64136130 0x7f9f641365a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c00c780 tx=0x7f9f5c00cc40 comp rx=0 tx=0).stop 2024-04-04T21:02:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 1 -- 172.21.15.106:0/246167929 shutdown_connections 2024-04-04T21:02:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 1 --2- 172.21.15.106:0/246167929 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f9f4806ceb0 0x7f9f4806f370 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 1 --2- 172.21.15.106:0/246167929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f64136130 0x7f9f641365a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 1 --2- 172.21.15.106:0/246167929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f6413f810 0x7f9f64139ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 1 --2- 172.21.15.106:0/246167929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f640a9b60 0x7f9f64139580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.464+0000 7f9f6c206640 1 -- 172.21.15.106:0/246167929 >> 172.21.15.106:0/246167929 conn(0x7f9f640b7ec0 msgr2=0x7f9f640b50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.465+0000 7f9f6c206640 1 -- 172.21.15.106:0/246167929 shutdown_connections 2024-04-04T21:02:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.465+0000 7f9f6c206640 1 -- 172.21.15.106:0/246167929 wait complete. 2024-04-04T21:02:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.465+0000 7f9f6c206640 1 librados: shutdown 2024-04-04T21:02:32.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.465+0000 7f5bd35c5640 10 librados: watch_flush enter 2024-04-04T21:02:32.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.465+0000 7f5bd35c5640 10 librados: watch_flush exit 2024-04-04T21:02:32.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.465+0000 7f5bd35c5640 20 client.4365.objecter shutdown clearing up homeless session... 2024-04-04T21:02:32.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.465+0000 7f5bd35c5640 10 client.4365.objecter successfully canceled tick 2024-04-04T21:02:32.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 1 -- 172.21.15.106:0/2769814709 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5bb006ce90 msgr2=0x7f5bb006f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 1 --2- 172.21.15.106:0/2769814709 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5bb006ce90 0x7f5bb006f350 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc144050 tx=0x7f5bbc006ba0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 10 monclient: shutdown 2024-04-04T21:02:32.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:32.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 1 -- 172.21.15.106:0/2769814709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5bcc1442f0 msgr2=0x7f5bcc143790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 1 --2- 172.21.15.106:0/2769814709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5bcc1442f0 0x7f5bcc143790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009870 tx=0x7f5bc80046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 1 -- 172.21.15.106:0/2769814709 shutdown_connections 2024-04-04T21:02:32.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 1 --2- 172.21.15.106:0/2769814709 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f5bb006ce90 0x7f5bb006f350 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 1 --2- 172.21.15.106:0/2769814709 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5bcc14a060 0x7f5bcc141570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 1 --2- 172.21.15.106:0/2769814709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc147150 0x7f5bcc141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 1 --2- 172.21.15.106:0/2769814709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5bcc1442f0 0x7f5bcc143790 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 1 -- 172.21.15.106:0/2769814709 >> 172.21.15.106:0/2769814709 conn(0x7f5bcc0b8030 msgr2=0x7f5bcc0b8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:32.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 1 -- 172.21.15.106:0/2769814709 shutdown_connections 2024-04-04T21:02:32.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 1 -- 172.21.15.106:0/2769814709 wait complete. 2024-04-04T21:02:32.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.466+0000 7f5bd35c5640 1 librados: shutdown 2024-04-04T21:02:32.483 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.6 2024-04-04T21:02:32.483 DEBUG:teuthology.parallel:result is None 2024-04-04T21:02:32.483 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345928 got 85899345929 for osd.3 2024-04-04T21:02:32.484 DEBUG:teuthology.parallel:result is None 2024-04-04T21:02:32.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.611+0000 7fdfa5e49640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-04T21:02:32.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.611+0000 7fdfa5e49640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-04T21:02:32.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.611+0000 7fdfa5e49640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:32.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.611+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2156704345 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fdfa00689d0 con 0x7fdfa00a9b60 2024-04-04T21:02:32.615 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.614+0000 7fdf87fff640 1 -- 172.21.15.106:0/2156704345 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdf9007ada0 con 0x7fdfa00a9b60 2024-04-04T21:02:32.615 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.614+0000 7fdf87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-04T21:02:32.615 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.614+0000 7fdf87fff640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:32.615 INFO:teuthology.orchestra.run.smithi106.stdout:85899345930 2024-04-04T21:02:32.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.616+0000 7fdfa5e49640 10 librados: watch_flush enter 2024-04-04T21:02:32.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.616+0000 7fdfa5e49640 10 librados: watch_flush exit 2024-04-04T21:02:32.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.616+0000 7fdfa5e49640 20 client.4374.objecter shutdown clearing up homeless session... 2024-04-04T21:02:32.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.616+0000 7fdfa5e49640 10 client.4374.objecter successfully canceled tick 2024-04-04T21:02:32.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.616+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2156704345 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdf8c06ce60 msgr2=0x7fdf8c06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.616+0000 7fdfa5e49640 1 --2- 172.21.15.106:0/2156704345 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdf8c06ce60 0x7fdf8c06f320 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00b5bb0 tx=0x7fdf94006ba0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.616+0000 7fdfa5e49640 10 monclient: shutdown 2024-04-04T21:02:32.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.616+0000 7fdfa5e49640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:32.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.616+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2156704345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfa00a9b60 msgr2=0x7fdfa0136d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.616+0000 7fdfa5e49640 1 --2- 172.21.15.106:0/2156704345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfa00a9b60 0x7fdfa0136d90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdf90000c00 tx=0x7fdf900046c0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.617+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2156704345 shutdown_connections 2024-04-04T21:02:32.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.617+0000 7fdfa5e49640 1 --2- 172.21.15.106:0/2156704345 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fdf8c06ce60 0x7fdf8c06f320 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.617+0000 7fdfa5e49640 1 --2- 172.21.15.106:0/2156704345 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdfa01372d0 0x7fdfa013a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.617+0000 7fdfa5e49640 1 --2- 172.21.15.106:0/2156704345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfa013f810 0x7fdfa013a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.618 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.617+0000 7fdfa5e49640 1 --2- 172.21.15.106:0/2156704345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdfa00a9b60 0x7fdfa0136d90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.617+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2156704345 >> 172.21.15.106:0/2156704345 conn(0x7fdfa00b7ec0 msgr2=0x7fdfa00b7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:32.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.617+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2156704345 shutdown_connections 2024-04-04T21:02:32.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.617+0000 7fdfa5e49640 1 -- 172.21.15.106:0/2156704345 wait complete. 2024-04-04T21:02:32.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.617+0000 7fdfa5e49640 1 librados: shutdown 2024-04-04T21:02:32.628 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345930 for osd.2 2024-04-04T21:02:32.628 DEBUG:teuthology.parallel:result is None 2024-04-04T21:02:32.628 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-04T21:02:32.628 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-04T21:02:32.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.726+0000 7fe9634e1640 10 monclient: get_monmap_and_config 2024-04-04T21:02:32.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.726+0000 7fe9634e1640 10 monclient: build_initial_monmap 2024-04-04T21:02:32.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.727+0000 7fe9634e1640 10 monclient: monmap: 2024-04-04T21:02:32.727 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:32.727 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:32.728 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:32.728070+0000 2024-04-04T21:02:32.728 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:32.728070+0000 2024-04-04T21:02:32.728 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:32.728 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:32.728 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:32.728 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:32.728 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:32.728 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:32.728 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:32.728 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:32.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.727+0000 7fe9634e1640 1 Processor -- start 2024-04-04T21:02:32.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.727+0000 7fe9634e1640 1 -- start start 2024-04-04T21:02:32.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.727+0000 7fe9634e1640 10 monclient: init 2024-04-04T21:02:32.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe9634e1640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:32.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe9634e1640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-04-04T21:02:32.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe9634e1640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe95c06c8c0 0x7fe95c0b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe9634e1640 10 monclient(hunting): picked mon.noname-a con 0x7fe95c06c8c0 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe9634e1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe95c145b50 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe9634e1640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe95c1466f0 0x7fe95c0afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe9634e1640 10 monclient(hunting): picked mon.noname-b con 0x7fe95c1466f0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe9634e1640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe95c0b0d00 con 0x7fe95c145b50 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe9634e1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe9634e1640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe95c0b0540 con 0x7fe95c06c8c0 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe9634e1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe9634e1640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c06d100 con 0x7fe95c1466f0 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe9634e1640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe9634e1640 10 monclient(hunting): authenticate will time out at 1401.995578s 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.728+0000 7fe961a7e640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe95c1466f0 0x7fe95c0afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.729+0000 7fe96127d640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe95c06c8c0 0x7fe95c0b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.729+0000 7fe960a7c640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fe95c145b50 0x7fe95c145fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:49620/0 (socket says 172.21.15.106:49620) 2024-04-04T21:02:32.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.729+0000 7fe960a7c640 1 -- 172.21.15.106:0/2338813775 learned_addr learned my addr 172.21.15.106:0/2338813775 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:32.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.729+0000 7fe961a7e640 10 monclient(hunting): get_auth_request con 0x7fe95c1466f0 auth_method 0 2024-04-04T21:02:32.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.729+0000 7fe961a7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:32.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.729+0000 7fe961a7e640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:32.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.729+0000 7fe961a7e640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:32.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.729+0000 7fe96127d640 10 monclient(hunting): get_auth_request con 0x7fe95c06c8c0 auth_method 0 2024-04-04T21:02:32.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.729+0000 7fe96127d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:32.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.729+0000 7fe96127d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:32.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.729+0000 7fe96127d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:32.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.729+0000 7fe961a7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:32.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.729+0000 7fe961a7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:32.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.729+0000 7fe961a7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:32.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.730+0000 7fe96127d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:32.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.730+0000 7fe96127d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:32.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.730+0000 7fe96127d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:32.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.731+0000 7fe961a7e640 10 monclient(hunting): handle_auth_done global_id 4358 payload 274 2024-04-04T21:02:32.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.731+0000 7fe961a7e640 1 -- 172.21.15.106:0/2338813775 >> v2:172.21.15.106:3300/0 conn(0x7fe95c06c8c0 msgr2=0x7fe95c0b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:32.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.731+0000 7fe961a7e640 1 --2- 172.21.15.106:0/2338813775 >> v2:172.21.15.106:3300/0 conn(0x7fe95c06c8c0 0x7fe95c0b07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.731+0000 7fe961a7e640 1 -- 172.21.15.106:0/2338813775 >> v1:172.21.15.106:6789/0 conn(0x7fe95c145b50 legacy=0x7fe95c145fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.731+0000 7fe961a7e640 10 monclient: _finish_hunting 0 2024-04-04T21:02:32.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.731+0000 7fe961a7e640 1 monclient: found mon.noname-b 2024-04-04T21:02:32.734 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.731+0000 7fe961a7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:32.734 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.731+0000 7fe961a7e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:32.734 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.731+0000 7fe961a7e640 1 -- 172.21.15.106:0/2338813775 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c0ae0d0 con 0x7fe95c1466f0 2024-04-04T21:02:32.734 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.733+0000 7fe961a7e640 1 --2- 172.21.15.106:0/2338813775 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe95c1466f0 0x7fe95c0afb30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe9540097a0 tx=0x7fe95402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f75d4443bae85787 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:32.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe953fff640 1 -- 172.21.15.106:0/2338813775 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9540315a0 con 0x7fe95c1466f0 2024-04-04T21:02:32.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:32.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe953fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:32.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe953fff640 10 monclient: dump: 2024-04-04T21:02:32.742 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:32.742 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:32.742 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:32.742 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:32.742 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:32.742 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:32.742 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:32.742 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:32.742 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:32.742 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:32.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe953fff640 10 monclient: _finish_auth 0 2024-04-04T21:02:32.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe953fff640 10 monclient: _check_auth_tickets 2024-04-04T21:02:32.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe953fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:32.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe953fff640 1 -- 172.21.15.106:0/2338813775 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe954037070 con 0x7fe95c1466f0 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe953fff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe953fff640 1 -- 172.21.15.106:0/2338813775 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe954031960 con 0x7fe95c1466f0 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe953fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe953fff640 10 monclient: dump: 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:32.743 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe9634e1640 5 monclient: authenticate success, global_id 4358 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe9634e1640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.736+0000 7fe9634e1640 4 set_mon_vals no callback set 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.737+0000 7fe9634e1640 10 monclient: shutdown 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.737+0000 7fe9634e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.737+0000 7fe9634e1640 1 -- 172.21.15.106:0/2338813775 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe95c1466f0 msgr2=0x7fe95c0afb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.737+0000 7fe9634e1640 1 --2- 172.21.15.106:0/2338813775 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe95c1466f0 0x7fe95c0afb30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe9540097a0 tx=0x7fe95402b840 comp rx=0 tx=0).stop 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.737+0000 7fe9634e1640 1 -- 172.21.15.106:0/2338813775 shutdown_connections 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.737+0000 7fe9634e1640 1 --2- 172.21.15.106:0/2338813775 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe95c1466f0 0x7fe95c0afb30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.737+0000 7fe9634e1640 1 --2- 172.21.15.106:0/2338813775 >> v2:172.21.15.106:3300/0 conn(0x7fe95c06c8c0 0x7fe95c0b07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.737+0000 7fe9634e1640 1 -- 172.21.15.106:0/2338813775 >> 172.21.15.106:0/2338813775 conn(0x7fe95c0b5fa0 msgr2=0x7fe95c0b63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.737+0000 7fe9634e1640 1 -- 172.21.15.106:0/2338813775 shutdown_connections 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.737+0000 7fe9634e1640 1 -- 172.21.15.106:0/2338813775 wait complete. 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.738+0000 7fe9634e1640 10 monclient: build_initial_monmap 2024-04-04T21:02:32.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.738+0000 7fe9634e1640 10 monclient: monmap: 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.738+0000 7fe9634e1640 1 librados: starting msgr at 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.738+0000 7fe9634e1640 1 librados: starting objecter 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.738+0000 7fe9634e1640 1 Processor -- start 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.738+0000 7fe9634e1640 1 -- start start 2024-04-04T21:02:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.738+0000 7fe9634e1640 1 librados: setting wanted keys 2024-04-04T21:02:32.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.738+0000 7fe9634e1640 1 librados: calling monclient init 2024-04-04T21:02:32.746 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.738+0000 7fe9634e1640 10 monclient: init 2024-04-04T21:02:32.747 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:32.747 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:02:32.747 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe95c06c8c0 0x7fe95c071bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:32.747 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 10 monclient(hunting): picked mon.noname-a con 0x7fe95c06c8c0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:32.747 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe95c145b50 0x7fe95c13b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:32.747 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 10 monclient(hunting): picked mon.noname-c con 0x7fe95c145b50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:32.747 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe95c1466f0 0x7fe95c13b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:32.748 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 10 monclient(hunting): picked mon.noname-b con 0x7fe95c1466f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:32.748 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:32.748 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c06dcc0 con 0x7fe95c06c8c0 2024-04-04T21:02:32.748 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:32.748 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c06db40 con 0x7fe95c1466f0 2024-04-04T21:02:32.748 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:32.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c06cca0 con 0x7fe95c145b50 2024-04-04T21:02:32.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:32.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe9634e1640 10 monclient(hunting): authenticate will time out at 1402.006567s 2024-04-04T21:02:32.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe960a7c640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe95c145b50 0x7fe95c13b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:32.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.739+0000 7fe96127d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe95c06c8c0 0x7fe95c071bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:32.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe961a7e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe95c1466f0 0x7fe95c13b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:32.749 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe96127d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe95c06c8c0 0x7fe95c071bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35182/0 (socket says 172.21.15.106:35182) 2024-04-04T21:02:32.750 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe96127d640 1 -- 172.21.15.106:0/3653725266 learned_addr learned my addr 172.21.15.106:0/3653725266 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:32.750 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe96127d640 10 monclient(hunting): get_auth_request con 0x7fe95c06c8c0 auth_method 0 2024-04-04T21:02:32.750 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe96127d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:32.750 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe96127d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:32.750 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe96127d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:32.750 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe960a7c640 10 monclient(hunting): get_auth_request con 0x7fe95c145b50 auth_method 0 2024-04-04T21:02:32.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe960a7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:32.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe960a7c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:32.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe960a7c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:32.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe961a7e640 10 monclient(hunting): get_auth_request con 0x7fe95c1466f0 auth_method 0 2024-04-04T21:02:32.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe961a7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:32.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe961a7e640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:32.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe961a7e640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:32.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe96127d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:32.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe96127d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:32.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe96127d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:32.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe960a7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:32.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe960a7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:32.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe960a7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:32.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe961a7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:32.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe961a7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:32.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.740+0000 7fe961a7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:32.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.743+0000 7fe96127d640 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2024-04-04T21:02:32.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.743+0000 7fe96127d640 1 -- 172.21.15.106:0/3653725266 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe95c145b50 msgr2=0x7fe95c13b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.743+0000 7fe96127d640 1 --2- 172.21.15.106:0/3653725266 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe95c145b50 0x7fe95c13b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.743+0000 7fe96127d640 1 -- 172.21.15.106:0/3653725266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe95c1466f0 msgr2=0x7fe95c13b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:32.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.743+0000 7fe96127d640 1 --2- 172.21.15.106:0/3653725266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe95c1466f0 0x7fe95c13b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:32.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.743+0000 7fe96127d640 10 monclient: _finish_hunting 0 2024-04-04T21:02:32.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.743+0000 7fe96127d640 1 monclient: found mon.noname-a 2024-04-04T21:02:32.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.743+0000 7fe96127d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:32.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.743+0000 7fe96127d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:32.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.743+0000 7fe96127d640 1 -- 172.21.15.106:0/3653725266 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c11f6d0 con 0x7fe95c06c8c0 2024-04-04T21:02:32.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.745+0000 7fe96127d640 1 --2- 172.21.15.106:0/3653725266 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe95c06c8c0 0x7fe95c071bd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe95800efc0 tx=0x7fe95800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:32.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.745+0000 7fe951ffb640 10 client.?.objecter ms_handle_connect 0x7fe95c06c8c0 2024-04-04T21:02:32.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.745+0000 7fe951ffb640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:32.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe951ffb640 1 -- 172.21.15.106:0/3653725266 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe958018020 con 0x7fe95c06c8c0 2024-04-04T21:02:32.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe951ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe951ffb640 10 monclient: dump: 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe951ffb640 10 monclient: _finish_auth 0 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe951ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:02:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe951ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe951ffb640 1 -- 172.21.15.106:0/3653725266 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe958014480 con 0x7fe95c06c8c0 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe951ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe951ffb640 1 -- 172.21.15.106:0/3653725266 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe958014850 con 0x7fe95c06c8c0 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe951ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe951ffb640 10 monclient: dump: 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:32.760 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:32.761 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:32.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe9634e1640 5 monclient: authenticate success, global_id 4383 2024-04-04T21:02:32.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe9634e1640 10 monclient: _renew_subs 2024-04-04T21:02:32.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe9634e1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:32.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe9634e1640 1 -- 172.21.15.106:0/3653725266 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95c13bea0 con 0x7fe95c06c8c0 2024-04-04T21:02:32.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe9537fe640 4 set_mon_vals no callback set 2024-04-04T21:02:32.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe9634e1640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:32.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe9634e1640 10 monclient: _renew_subs 2024-04-04T21:02:32.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe9634e1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:32.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe9634e1640 1 -- 172.21.15.106:0/3653725266 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95c13c130 con 0x7fe95c06c8c0 2024-04-04T21:02:32.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.749+0000 7fe9634e1640 1 librados: init done 2024-04-04T21:02:32.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.750+0000 7fe9634e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:32.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.750+0000 7fe9634e1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:32.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.750+0000 7fe9634e1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:32.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.750+0000 7fe9634e1640 1 -- 172.21.15.106:0/3653725266 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe95c04f050 con 0x7fe95c06c8c0 2024-04-04T21:02:32.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.768+0000 7fe951ffb640 1 -- 172.21.15.106:0/3653725266 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe958005d10 con 0x7fe95c06c8c0 2024-04-04T21:02:32.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.768+0000 7fe951ffb640 1 --2- 172.21.15.106:0/3653725266 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe94006cee0 0x7fe94006f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:32.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.768+0000 7fe960a7c640 1 --2- 172.21.15.106:0/3653725266 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe94006cee0 0x7fe94006f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:32.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.768+0000 7fe960a7c640 10 monclient: get_auth_request con 0x7fe94006cee0 auth_method 0 2024-04-04T21:02:32.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.768+0000 7fe951ffb640 10 client.4383.objecter ms_handle_connect 0x7fe94006cee0 2024-04-04T21:02:32.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.768+0000 7fe960a7c640 1 --2- 172.21.15.106:0/3653725266 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe94006cee0 0x7fe94006f3a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe95c13cdd0 tx=0x7fe94c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:32.772 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.771+0000 7fe951ffb640 1 -- 172.21.15.106:0/3653725266 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe958057cd0 con 0x7fe95c06c8c0 2024-04-04T21:02:32.772 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.771+0000 7fe951ffb640 10 client.4383.objecter ms_dispatch 0x7fe95c000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:32.772 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.771+0000 7fe951ffb640 3 client.4383.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:32.772 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.771+0000 7fe951ffb640 3 client.4383.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:32.772 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:32.771+0000 7fe951ffb640 20 client.4383.objecter dump_active .. 0 homeless 2024-04-04T21:02:33.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.012+0000 7fe951ffb640 1 -- 172.21.15.106:0/3653725266 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe958022070 con 0x7fe95c06c8c0 2024-04-04T21:02:33.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.012+0000 7fe951ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:33.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.012+0000 7fe951ffb640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:33.193 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.192+0000 7fe9634e1640 1 -- 172.21.15.106:0/3653725266 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe95c06c240 con 0x7fe94006cee0 2024-04-04T21:02:33.194 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.194+0000 7fe951ffb640 1 -- 172.21.15.106:0/3653725266 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+35558 (secure 0 0 0) 0x7fe95c06c240 con 0x7fe94006cee0 2024-04-04T21:02:33.195 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:02:33.195 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-04-04T21:02:33.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.195+0000 7fe9634e1640 10 librados: watch_flush enter 2024-04-04T21:02:33.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.195+0000 7fe9634e1640 10 librados: watch_flush exit 2024-04-04T21:02:33.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.195+0000 7fe9634e1640 20 client.4383.objecter shutdown clearing up homeless session... 2024-04-04T21:02:33.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.195+0000 7fe9634e1640 10 client.4383.objecter successfully canceled tick 2024-04-04T21:02:33.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 1 -- 172.21.15.106:0/3653725266 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe94006cee0 msgr2=0x7fe94006f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:33.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 1 --2- 172.21.15.106:0/3653725266 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe94006cee0 0x7fe94006f3a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe95c13cdd0 tx=0x7fe94c009040 comp rx=0 tx=0).stop 2024-04-04T21:02:33.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 10 monclient: shutdown 2024-04-04T21:02:33.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:33.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 1 -- 172.21.15.106:0/3653725266 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe95c06c8c0 msgr2=0x7fe95c071bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:33.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 1 --2- 172.21.15.106:0/3653725266 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe95c06c8c0 0x7fe95c071bd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe95800efc0 tx=0x7fe95800c490 comp rx=0 tx=0).stop 2024-04-04T21:02:33.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 1 -- 172.21.15.106:0/3653725266 shutdown_connections 2024-04-04T21:02:33.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 1 --2- 172.21.15.106:0/3653725266 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe94006cee0 0x7fe94006f3a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 1 --2- 172.21.15.106:0/3653725266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe95c1466f0 0x7fe95c13b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 1 --2- 172.21.15.106:0/3653725266 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe95c145b50 0x7fe95c13b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.198 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 1 --2- 172.21.15.106:0/3653725266 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe95c06c8c0 0x7fe95c071bd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.198 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 1 -- 172.21.15.106:0/3653725266 >> 172.21.15.106:0/3653725266 conn(0x7fe95c0b5fa0 msgr2=0x7fe95c0af250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:33.198 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 1 -- 172.21.15.106:0/3653725266 shutdown_connections 2024-04-04T21:02:33.198 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 1 -- 172.21.15.106:0/3653725266 wait complete. 2024-04-04T21:02:33.198 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.196+0000 7fe9634e1640 1 librados: shutdown 2024-04-04T21:02:33.207 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-04-04T21:02:32.379989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218984,"kb_used_data":2808,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755736,"statfs":{"total":773094113280,"available":772869873664,"internally_reserved":0,"allocated":2875392,"data_stored":1557204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":70,"apply_latency_ms":70,"commit_latency_ns":70000000,"apply_latency_ns":70000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.333186"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.405532+0000","last_change":"2024-04-04T21:02:25.403337+0000","last_active":"2024-04-04T21:02:27.405532+0000","last_peered":"2024-04-04T21:02:27.405532+0000","last_clean":"2024-04-04T21:02:27.405532+0000","last_became_active":"2024-04-04T21:02:25.394276+0000","last_became_peered":"2024-04-04T21:02:25.394276+0000","last_unstale":"2024-04-04T21:02:27.405532+0000","last_undegraded":"2024-04-04T21:02:27.405532+0000","last_fullsized":"2024-04-04T21:02:27.405532+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:42:02.653040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.416342+0000","last_change":"2024-04-04T21:02:25.438251+0000","last_active":"2024-04-04T21:02:27.416342+0000","last_peered":"2024-04-04T21:02:27.416342+0000","last_clean":"2024-04-04T21:02:27.416342+0000","last_became_active":"2024-04-04T21:02:25.412507+0000","last_became_peered":"2024-04-04T21:02:25.412507+0000","last_unstale":"2024-04-04T21:02:27.416342+0000","last_undegraded":"2024-04-04T21:02:27.416342+0000","last_fullsized":"2024-04-04T21:02:27.416342+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:17:04.580586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.409204+0000","last_change":"2024-04-04T21:02:24.241805+0000","last_active":"2024-04-04T21:02:27.409204+0000","last_peered":"2024-04-04T21:02:27.409204+0000","last_clean":"2024-04-04T21:02:27.409204+0000","last_became_active":"2024-04-04T21:02:24.236662+0000","last_became_peered":"2024-04-04T21:02:24.236662+0000","last_unstale":"2024-04-04T21:02:27.409204+0000","last_undegraded":"2024-04-04T21:02:27.409204+0000","last_fullsized":"2024-04-04T21:02:27.409204+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:47:33.277018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.308760+0000","last_change":"2024-04-04T21:02:24.374143+0000","last_active":"2024-04-04T21:02:27.308760+0000","last_peered":"2024-04-04T21:02:27.308760+0000","last_clean":"2024-04-04T21:02:27.308760+0000","last_became_active":"2024-04-04T21:02:24.359409+0000","last_became_peered":"2024-04-04T21:02:24.359409+0000","last_unstale":"2024-04-04T21:02:27.308760+0000","last_undegraded":"2024-04-04T21:02:27.308760+0000","last_fullsized":"2024-04-04T21:02:27.308760+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:43:28.396677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.398795+0000","last_change":"2024-04-04T21:02:24.359272+0000","last_active":"2024-04-04T21:02:27.398795+0000","last_peered":"2024-04-04T21:02:27.398795+0000","last_clean":"2024-04-04T21:02:27.398795+0000","last_became_active":"2024-04-04T21:02:24.352103+0000","last_became_peered":"2024-04-04T21:02:24.352103+0000","last_unstale":"2024-04-04T21:02:27.398795+0000","last_undegraded":"2024-04-04T21:02:27.398795+0000","last_fullsized":"2024-04-04T21:02:27.398795+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:36:54.774282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.375450+0000","last_change":"2024-04-04T21:02:24.189161+0000","last_active":"2024-04-04T21:02:27.375450+0000","last_peered":"2024-04-04T21:02:27.375450+0000","last_clean":"2024-04-04T21:02:27.375450+0000","last_became_active":"2024-04-04T21:02:24.159102+0000","last_became_peered":"2024-04-04T21:02:24.159102+0000","last_unstale":"2024-04-04T21:02:27.375450+0000","last_undegraded":"2024-04-04T21:02:27.375450+0000","last_fullsized":"2024-04-04T21:02:27.375450+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:02:24.748876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.382972+0000","last_change":"2024-04-04T21:02:24.174900+0000","last_active":"2024-04-04T21:02:27.382972+0000","last_peered":"2024-04-04T21:02:27.382972+0000","last_clean":"2024-04-04T21:02:27.382972+0000","last_became_active":"2024-04-04T21:02:24.146676+0000","last_became_peered":"2024-04-04T21:02:24.146676+0000","last_unstale":"2024-04-04T21:02:27.382972+0000","last_undegraded":"2024-04-04T21:02:27.382972+0000","last_fullsized":"2024-04-04T21:02:27.382972+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:47:28.827956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.402219+0000","last_change":"2024-04-04T21:02:24.476744+0000","last_active":"2024-04-04T21:02:27.402219+0000","last_peered":"2024-04-04T21:02:27.402219+0000","last_clean":"2024-04-04T21:02:27.402219+0000","last_became_active":"2024-04-04T21:02:24.466203+0000","last_became_peered":"2024-04-04T21:02:24.466203+0000","last_unstale":"2024-04-04T21:02:27.402219+0000","last_undegraded":"2024-04-04T21:02:27.402219+0000","last_fullsized":"2024-04-04T21:02:27.402219+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T21:40:50.021063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":26,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.363811+0000","last_change":"2024-04-04T21:02:20.509950+0000","last_active":"2024-04-04T21:02:27.363811+0000","last_peered":"2024-04-04T21:02:27.363811+0000","last_clean":"2024-04-04T21:02:27.363811+0000","last_became_active":"2024-04-04T21:02:19.876220+0000","last_became_peered":"2024-04-04T21:02:19.876220+0000","last_unstale":"2024-04-04T21:02:27.363811+0000","last_undegraded":"2024-04-04T21:02:27.363811+0000","last_fullsized":"2024-04-04T21:02:27.363811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:05.463818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:05.463818+0000","last_clean_scrub_stamp":"2024-04-04T21:02:05.463818+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:18:20.734360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345929,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":91133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":96543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899345930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":651264,"data_stored":488973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":96543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509450,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":96543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":96543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":96543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":663552,"data_stored":494383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-04T21:02:33.208 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-04T21:02:33.309 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.308+0000 7ffbf13c1640 10 monclient: get_monmap_and_config 2024-04-04T21:02:33.309 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.308+0000 7ffbf13c1640 10 monclient: build_initial_monmap 2024-04-04T21:02:33.309 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.308+0000 7ffbf13c1640 10 monclient: monmap: 2024-04-04T21:02:33.309 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:33.309 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:33.309 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:33.309208+0000 2024-04-04T21:02:33.309 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:33.309208+0000 2024-04-04T21:02:33.309 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:33.309 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:33.309 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:33.309 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:33.309 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:33.309 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:33.310 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:33.310 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:33.310 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.308+0000 7ffbf13c1640 1 Processor -- start 2024-04-04T21:02:33.310 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.308+0000 7ffbf13c1640 1 -- start start 2024-04-04T21:02:33.310 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.308+0000 7ffbf13c1640 10 monclient: init 2024-04-04T21:02:33.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:33.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ffbec0a9a50 0x7ffbec0a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 10 monclient(hunting): picked mon.noname-a con 0x7ffbec0a9a50 addr v2:172.21.15.106:3300/0 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ffbec13f6a0 0x7ffbec13fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 10 monclient(hunting): picked mon.noname-c con 0x7ffbec13f6a0 addr v2:172.21.15.203:3300/0 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbec142880 0x7ffbec142d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 10 monclient(hunting): picked mon.noname-b con 0x7ffbec142880 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbec0aa370 con 0x7ffbec0a9a50 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbec0ab750 con 0x7ffbec13f6a0 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec140120 con 0x7ffbec142880 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbf13c1640 10 monclient(hunting): authenticate will time out at 1402.576499s 2024-04-04T21:02:33.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbeb7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbec142880 0x7ffbec142d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:33.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.309+0000 7ffbeaffd640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ffbec0a9a50 0x7ffbec0a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:33.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbea7fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ffbec13f6a0 0x7ffbec13fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:33.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeaffd640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ffbec0a9a50 0x7ffbec0a9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35190/0 (socket says 172.21.15.106:35190) 2024-04-04T21:02:33.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeaffd640 1 -- 172.21.15.106:0/3170104144 learned_addr learned my addr 172.21.15.106:0/3170104144 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:33.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeaffd640 10 monclient(hunting): get_auth_request con 0x7ffbec0a9a50 auth_method 0 2024-04-04T21:02:33.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:33.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeaffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:33.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:33.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeb7fe640 10 monclient(hunting): get_auth_request con 0x7ffbec142880 auth_method 0 2024-04-04T21:02:33.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:33.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:33.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:33.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbea7fc640 10 monclient(hunting): get_auth_request con 0x7ffbec13f6a0 auth_method 0 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbea7fc640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbeb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.310+0000 7ffbea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.312+0000 7ffbeaffd640 10 monclient(hunting): handle_auth_done global_id 4386 payload 274 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.312+0000 7ffbeaffd640 1 -- 172.21.15.106:0/3170104144 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbec142880 msgr2=0x7ffbec142d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:33.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.312+0000 7ffbeaffd640 1 --2- 172.21.15.106:0/3170104144 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbec142880 0x7ffbec142d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.312+0000 7ffbeaffd640 1 -- 172.21.15.106:0/3170104144 >> v2:172.21.15.203:3300/0 conn(0x7ffbec13f6a0 msgr2=0x7ffbec13fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:02:33.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.312+0000 7ffbeaffd640 1 --2- 172.21.15.106:0/3170104144 >> v2:172.21.15.203:3300/0 conn(0x7ffbec13f6a0 0x7ffbec13fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.312+0000 7ffbeaffd640 10 monclient: _finish_hunting 0 2024-04-04T21:02:33.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.312+0000 7ffbeaffd640 1 monclient: found mon.noname-a 2024-04-04T21:02:33.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.312+0000 7ffbeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:33.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.312+0000 7ffbeaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:33.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.312+0000 7ffbeaffd640 1 -- 172.21.15.106:0/3170104144 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbec0aa660 con 0x7ffbec0a9a50 2024-04-04T21:02:33.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.312+0000 7ffbeb7fe640 1 --2- 172.21.15.106:0/3170104144 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbec142880 0x7ffbec142d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-04T21:02:33.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.314+0000 7ffbeaffd640 1 --2- 172.21.15.106:0/3170104144 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbec0a9a50 0x7ffbec0a9e30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffbec06c390 tx=0x7ffbe402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dadd00015ae635ec server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:33.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbe9ffb640 1 -- 172.21.15.106:0/3170104144 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe40317d0 con 0x7ffbec0a9a50 2024-04-04T21:02:33.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbe9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:33.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbe9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:33.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbe9ffb640 10 monclient: dump: 2024-04-04T21:02:33.320 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:33.320 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:33.320 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.320 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.320 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:33.320 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:33.321 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:33.321 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:33.321 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:33.321 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:33.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbe9ffb640 10 monclient: _finish_auth 0 2024-04-04T21:02:33.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbe9ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:02:33.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbe9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:33.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbe9ffb640 1 -- 172.21.15.106:0/3170104144 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe4037070 con 0x7ffbec0a9a50 2024-04-04T21:02:33.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbe9ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:33.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbe9ffb640 1 -- 172.21.15.106:0/3170104144 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe4031b90 con 0x7ffbec0a9a50 2024-04-04T21:02:33.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbf13c1640 5 monclient: authenticate success, global_id 4386 2024-04-04T21:02:33.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbf13c1640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:33.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbf13c1640 4 set_mon_vals no callback set 2024-04-04T21:02:33.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbe9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbe9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbe9ffb640 10 monclient: dump: 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbf13c1640 10 monclient: shutdown 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbf13c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:33.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbf13c1640 1 -- 172.21.15.106:0/3170104144 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbec0a9a50 msgr2=0x7ffbec0a9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:33.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbf13c1640 1 --2- 172.21.15.106:0/3170104144 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbec0a9a50 0x7ffbec0a9e30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffbec06c390 tx=0x7ffbe402b870 comp rx=0 tx=0).stop 2024-04-04T21:02:33.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbf13c1640 1 -- 172.21.15.106:0/3170104144 shutdown_connections 2024-04-04T21:02:33.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbf13c1640 1 --2- 172.21.15.106:0/3170104144 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbec142880 0x7ffbec142d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.318+0000 7ffbf13c1640 1 --2- 172.21.15.106:0/3170104144 >> v2:172.21.15.203:3300/0 conn(0x7ffbec13f6a0 0x7ffbec13fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.319+0000 7ffbf13c1640 1 --2- 172.21.15.106:0/3170104144 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbec0a9a50 0x7ffbec0a9e30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.319+0000 7ffbf13c1640 1 -- 172.21.15.106:0/3170104144 >> 172.21.15.106:0/3170104144 conn(0x7ffbec0b8030 msgr2=0x7ffbec0b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:33.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.319+0000 7ffbf13c1640 1 -- 172.21.15.106:0/3170104144 shutdown_connections 2024-04-04T21:02:33.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.319+0000 7ffbf13c1640 1 -- 172.21.15.106:0/3170104144 wait complete. 2024-04-04T21:02:33.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.319+0000 7ffbf13c1640 10 monclient: build_initial_monmap 2024-04-04T21:02:33.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.319+0000 7ffbf13c1640 10 monclient: monmap: 2024-04-04T21:02:33.327 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:33.327 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:33.327 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:33.327 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:33.327 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:33.327 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:33.327 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:33.327 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:33.327 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:33.327 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:33.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 1 librados: starting msgr at 2024-04-04T21:02:33.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 1 librados: starting objecter 2024-04-04T21:02:33.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 1 Processor -- start 2024-04-04T21:02:33.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 1 -- start start 2024-04-04T21:02:33.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 1 librados: setting wanted keys 2024-04-04T21:02:33.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 1 librados: calling monclient init 2024-04-04T21:02:33.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 10 monclient: init 2024-04-04T21:02:33.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:33.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:02:33.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbec0a9a50 0x7ffbec1db930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:33.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 10 monclient(hunting): picked mon.noname-c con 0x7ffbec0a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:33.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbec13f6a0 0x7ffbec1dbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:33.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 10 monclient(hunting): picked mon.noname-b con 0x7ffbec13f6a0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:33.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbec142880 0x7ffbec1d8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:33.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 10 monclient(hunting): picked mon.noname-a con 0x7ffbec142880 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:33.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:33.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec145b60 con 0x7ffbec142880 2024-04-04T21:02:33.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:33.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec143460 con 0x7ffbec13f6a0 2024-04-04T21:02:33.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:33.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec145ce0 con 0x7ffbec0a9a50 2024-04-04T21:02:33.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:33.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.320+0000 7ffbf13c1640 10 monclient(hunting): authenticate will time out at 1402.588131s 2024-04-04T21:02:33.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.321+0000 7ffbeb7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbec142880 0x7ffbec1d8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:33.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.321+0000 7ffbeaffd640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbec0a9a50 0x7ffbec1db930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:33.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.321+0000 7ffbeb7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbec142880 0x7ffbec1d8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35206/0 (socket says 172.21.15.106:35206) 2024-04-04T21:02:33.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.321+0000 7ffbeb7fe640 1 -- 172.21.15.106:0/159101558 learned_addr learned my addr 172.21.15.106:0/159101558 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:33.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.321+0000 7ffbeb7fe640 10 monclient(hunting): get_auth_request con 0x7ffbec142880 auth_method 0 2024-04-04T21:02:33.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.321+0000 7ffbeb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:33.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.321+0000 7ffbeb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.321+0000 7ffbeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.321+0000 7ffbea7fc640 1 --2- 172.21.15.106:0/159101558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbec13f6a0 0x7ffbec1dbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.321+0000 7ffbeaffd640 10 monclient(hunting): get_auth_request con 0x7ffbec0a9a50 auth_method 0 2024-04-04T21:02:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.321+0000 7ffbeaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.321+0000 7ffbeaffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.321+0000 7ffbeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.322+0000 7ffbea7fc640 10 monclient(hunting): get_auth_request con 0x7ffbec13f6a0 auth_method 0 2024-04-04T21:02:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.322+0000 7ffbea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.322+0000 7ffbea7fc640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.322+0000 7ffbea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.322+0000 7ffbeb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.322+0000 7ffbeb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.322+0000 7ffbeb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.322+0000 7ffbeaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:33.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.322+0000 7ffbeaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:33.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.322+0000 7ffbeaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:33.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.322+0000 7ffbea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:33.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.322+0000 7ffbea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:33.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.322+0000 7ffbea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:33.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.325+0000 7ffbeb7fe640 10 monclient(hunting): handle_auth_done global_id 4389 payload 824 2024-04-04T21:02:33.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.325+0000 7ffbeb7fe640 1 -- 172.21.15.106:0/159101558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbec0a9a50 msgr2=0x7ffbec1db930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:33.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.325+0000 7ffbeb7fe640 1 --2- 172.21.15.106:0/159101558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbec0a9a50 0x7ffbec1db930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.325+0000 7ffbeb7fe640 1 -- 172.21.15.106:0/159101558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbec13f6a0 msgr2=0x7ffbec1dbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:33.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.325+0000 7ffbeb7fe640 1 --2- 172.21.15.106:0/159101558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbec13f6a0 0x7ffbec1dbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.325+0000 7ffbeb7fe640 10 monclient: _finish_hunting 0 2024-04-04T21:02:33.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.325+0000 7ffbeb7fe640 1 monclient: found mon.noname-a 2024-04-04T21:02:33.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.325+0000 7ffbeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:33.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.325+0000 7ffbeb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:33.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.325+0000 7ffbeb7fe640 1 -- 172.21.15.106:0/159101558 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbec0aebd0 con 0x7ffbec142880 2024-04-04T21:02:33.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.325+0000 7ffbeaffd640 1 --2- 172.21.15.106:0/159101558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbec0a9a50 0x7ffbec1db930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-04T21:02:33.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.327+0000 7ffbeb7fe640 1 --2- 172.21.15.106:0/159101558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbec142880 0x7ffbec1d8180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00e8b0 tx=0x7ffbdc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:33.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.327+0000 7ffbd37fe640 10 client.?.objecter ms_handle_connect 0x7ffbec142880 2024-04-04T21:02:33.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.327+0000 7ffbd37fe640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:33.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbd37fe640 1 -- 172.21.15.106:0/159101558 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc017070 con 0x7ffbec142880 2024-04-04T21:02:33.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:33.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbd37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:33.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbd37fe640 10 monclient: dump: 2024-04-04T21:02:33.334 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:33.334 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:33.334 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.334 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.334 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:33.334 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:33.334 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:33.334 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:33.334 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:33.334 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:33.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbd37fe640 10 monclient: _finish_auth 0 2024-04-04T21:02:33.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbd37fe640 10 monclient: _check_auth_tickets 2024-04-04T21:02:33.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:33.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbd37fe640 1 -- 172.21.15.106:0/159101558 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbdc004020 con 0x7ffbec142880 2024-04-04T21:02:33.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbd37fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:33.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbd37fe640 1 -- 172.21.15.106:0/159101558 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc0043f0 con 0x7ffbec142880 2024-04-04T21:02:33.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbf13c1640 5 monclient: authenticate success, global_id 4389 2024-04-04T21:02:33.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbf13c1640 10 monclient: _renew_subs 2024-04-04T21:02:33.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbf13c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:33.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbf13c1640 1 -- 172.21.15.106:0/159101558 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbec144090 con 0x7ffbec142880 2024-04-04T21:02:33.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbd37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbd37fe640 10 monclient: dump: 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbe97fa640 4 set_mon_vals no callback set 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbf13c1640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbf13c1640 10 monclient: _renew_subs 2024-04-04T21:02:33.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbf13c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:33.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbf13c1640 1 -- 172.21.15.106:0/159101558 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbec1d88a0 con 0x7ffbec142880 2024-04-04T21:02:33.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.331+0000 7ffbf13c1640 1 librados: init done 2024-04-04T21:02:33.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.332+0000 7ffbf13c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:33.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.332+0000 7ffbf13c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:33.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.332+0000 7ffbf13c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:33.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.332+0000 7ffbf13c1640 1 -- 172.21.15.106:0/159101558 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbec0aa130 con 0x7ffbec142880 2024-04-04T21:02:33.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.582+0000 7ffbd37fe640 1 -- 172.21.15.106:0/159101558 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ffbdc00c940 con 0x7ffbec142880 2024-04-04T21:02:33.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.583+0000 7ffbd37fe640 1 --2- 172.21.15.106:0/159101558 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ffbd806ce60 0x7ffbd806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:33.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.583+0000 7ffbeaffd640 1 --2- 172.21.15.106:0/159101558 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ffbd806ce60 0x7ffbd806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:33.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.583+0000 7ffbeaffd640 10 monclient: get_auth_request con 0x7ffbd806ce60 auth_method 0 2024-04-04T21:02:33.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.583+0000 7ffbd37fe640 10 client.4389.objecter ms_handle_connect 0x7ffbd806ce60 2024-04-04T21:02:33.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.583+0000 7ffbeaffd640 1 --2- 172.21.15.106:0/159101558 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ffbd806ce60 0x7ffbd806f320 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4036000 tx=0x7ffbe403d040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:33.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.597+0000 7ffbd37fe640 1 -- 172.21.15.106:0/159101558 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ffbdc056f90 con 0x7ffbec142880 2024-04-04T21:02:33.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.597+0000 7ffbd37fe640 10 client.4389.objecter ms_dispatch 0x7ffbec000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:33.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.597+0000 7ffbd37fe640 3 client.4389.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:33.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.597+0000 7ffbd37fe640 3 client.4389.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:33.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.597+0000 7ffbd37fe640 20 client.4389.objecter dump_active .. 0 homeless 2024-04-04T21:02:33.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.597+0000 7ffbd37fe640 1 -- 172.21.15.106:0/159101558 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ffbec0aa130 con 0x7ffbec142880 2024-04-04T21:02:33.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.597+0000 7ffbd37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:33.599 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.597+0000 7ffbd37fe640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:33.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.774+0000 7ffbf13c1640 1 -- 172.21.15.106:0/159101558 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffbec04f050 con 0x7ffbd806ce60 2024-04-04T21:02:33.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.776+0000 7ffbd37fe640 1 -- 172.21.15.106:0/159101558 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+35558 (secure 0 0 0) 0x7ffbec04f050 con 0x7ffbd806ce60 2024-04-04T21:02:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:02:33.777 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-04-04T21:02:33.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.777+0000 7ffbf13c1640 10 librados: watch_flush enter 2024-04-04T21:02:33.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 10 librados: watch_flush exit 2024-04-04T21:02:33.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 20 client.4389.objecter shutdown clearing up homeless session... 2024-04-04T21:02:33.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 10 client.4389.objecter successfully canceled tick 2024-04-04T21:02:33.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 1 -- 172.21.15.106:0/159101558 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ffbd806ce60 msgr2=0x7ffbd806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:33.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 1 --2- 172.21.15.106:0/159101558 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ffbd806ce60 0x7ffbd806f320 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4036000 tx=0x7ffbe403d040 comp rx=0 tx=0).stop 2024-04-04T21:02:33.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 10 monclient: shutdown 2024-04-04T21:02:33.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:33.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 1 -- 172.21.15.106:0/159101558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbec142880 msgr2=0x7ffbec1d8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:33.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 1 --2- 172.21.15.106:0/159101558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbec142880 0x7ffbec1d8180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00e8b0 tx=0x7ffbdc00ed70 comp rx=0 tx=0).stop 2024-04-04T21:02:33.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 1 -- 172.21.15.106:0/159101558 shutdown_connections 2024-04-04T21:02:33.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 1 --2- 172.21.15.106:0/159101558 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ffbd806ce60 0x7ffbd806f320 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 1 --2- 172.21.15.106:0/159101558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbec142880 0x7ffbec1d8180 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 1 --2- 172.21.15.106:0/159101558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbec13f6a0 0x7ffbec1dbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 1 --2- 172.21.15.106:0/159101558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbec0a9a50 0x7ffbec1db930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 1 -- 172.21.15.106:0/159101558 >> 172.21.15.106:0/159101558 conn(0x7ffbec0b8030 msgr2=0x7ffbec0b8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:33.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 1 -- 172.21.15.106:0/159101558 shutdown_connections 2024-04-04T21:02:33.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 1 -- 172.21.15.106:0/159101558 wait complete. 2024-04-04T21:02:33.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.778+0000 7ffbf13c1640 1 librados: shutdown 2024-04-04T21:02:33.789 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-04-04T21:02:32.379989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218984,"kb_used_data":2808,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755736,"statfs":{"total":773094113280,"available":772869873664,"internally_reserved":0,"allocated":2875392,"data_stored":1557204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":70,"apply_latency_ms":70,"commit_latency_ns":70000000,"apply_latency_ns":70000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.333186"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.405532+0000","last_change":"2024-04-04T21:02:25.403337+0000","last_active":"2024-04-04T21:02:27.405532+0000","last_peered":"2024-04-04T21:02:27.405532+0000","last_clean":"2024-04-04T21:02:27.405532+0000","last_became_active":"2024-04-04T21:02:25.394276+0000","last_became_peered":"2024-04-04T21:02:25.394276+0000","last_unstale":"2024-04-04T21:02:27.405532+0000","last_undegraded":"2024-04-04T21:02:27.405532+0000","last_fullsized":"2024-04-04T21:02:27.405532+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:42:02.653040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.416342+0000","last_change":"2024-04-04T21:02:25.438251+0000","last_active":"2024-04-04T21:02:27.416342+0000","last_peered":"2024-04-04T21:02:27.416342+0000","last_clean":"2024-04-04T21:02:27.416342+0000","last_became_active":"2024-04-04T21:02:25.412507+0000","last_became_peered":"2024-04-04T21:02:25.412507+0000","last_unstale":"2024-04-04T21:02:27.416342+0000","last_undegraded":"2024-04-04T21:02:27.416342+0000","last_fullsized":"2024-04-04T21:02:27.416342+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:17:04.580586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.409204+0000","last_change":"2024-04-04T21:02:24.241805+0000","last_active":"2024-04-04T21:02:27.409204+0000","last_peered":"2024-04-04T21:02:27.409204+0000","last_clean":"2024-04-04T21:02:27.409204+0000","last_became_active":"2024-04-04T21:02:24.236662+0000","last_became_peered":"2024-04-04T21:02:24.236662+0000","last_unstale":"2024-04-04T21:02:27.409204+0000","last_undegraded":"2024-04-04T21:02:27.409204+0000","last_fullsized":"2024-04-04T21:02:27.409204+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:47:33.277018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.308760+0000","last_change":"2024-04-04T21:02:24.374143+0000","last_active":"2024-04-04T21:02:27.308760+0000","last_peered":"2024-04-04T21:02:27.308760+0000","last_clean":"2024-04-04T21:02:27.308760+0000","last_became_active":"2024-04-04T21:02:24.359409+0000","last_became_peered":"2024-04-04T21:02:24.359409+0000","last_unstale":"2024-04-04T21:02:27.308760+0000","last_undegraded":"2024-04-04T21:02:27.308760+0000","last_fullsized":"2024-04-04T21:02:27.308760+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:43:28.396677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.398795+0000","last_change":"2024-04-04T21:02:24.359272+0000","last_active":"2024-04-04T21:02:27.398795+0000","last_peered":"2024-04-04T21:02:27.398795+0000","last_clean":"2024-04-04T21:02:27.398795+0000","last_became_active":"2024-04-04T21:02:24.352103+0000","last_became_peered":"2024-04-04T21:02:24.352103+0000","last_unstale":"2024-04-04T21:02:27.398795+0000","last_undegraded":"2024-04-04T21:02:27.398795+0000","last_fullsized":"2024-04-04T21:02:27.398795+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:36:54.774282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.375450+0000","last_change":"2024-04-04T21:02:24.189161+0000","last_active":"2024-04-04T21:02:27.375450+0000","last_peered":"2024-04-04T21:02:27.375450+0000","last_clean":"2024-04-04T21:02:27.375450+0000","last_became_active":"2024-04-04T21:02:24.159102+0000","last_became_peered":"2024-04-04T21:02:24.159102+0000","last_unstale":"2024-04-04T21:02:27.375450+0000","last_undegraded":"2024-04-04T21:02:27.375450+0000","last_fullsized":"2024-04-04T21:02:27.375450+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:02:24.748876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.382972+0000","last_change":"2024-04-04T21:02:24.174900+0000","last_active":"2024-04-04T21:02:27.382972+0000","last_peered":"2024-04-04T21:02:27.382972+0000","last_clean":"2024-04-04T21:02:27.382972+0000","last_became_active":"2024-04-04T21:02:24.146676+0000","last_became_peered":"2024-04-04T21:02:24.146676+0000","last_unstale":"2024-04-04T21:02:27.382972+0000","last_undegraded":"2024-04-04T21:02:27.382972+0000","last_fullsized":"2024-04-04T21:02:27.382972+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:47:28.827956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.402219+0000","last_change":"2024-04-04T21:02:24.476744+0000","last_active":"2024-04-04T21:02:27.402219+0000","last_peered":"2024-04-04T21:02:27.402219+0000","last_clean":"2024-04-04T21:02:27.402219+0000","last_became_active":"2024-04-04T21:02:24.466203+0000","last_became_peered":"2024-04-04T21:02:24.466203+0000","last_unstale":"2024-04-04T21:02:27.402219+0000","last_undegraded":"2024-04-04T21:02:27.402219+0000","last_fullsized":"2024-04-04T21:02:27.402219+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T21:40:50.021063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":26,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-04T21:02:27.363811+0000","last_change":"2024-04-04T21:02:20.509950+0000","last_active":"2024-04-04T21:02:27.363811+0000","last_peered":"2024-04-04T21:02:27.363811+0000","last_clean":"2024-04-04T21:02:27.363811+0000","last_became_active":"2024-04-04T21:02:19.876220+0000","last_became_peered":"2024-04-04T21:02:19.876220+0000","last_unstale":"2024-04-04T21:02:27.363811+0000","last_undegraded":"2024-04-04T21:02:27.363811+0000","last_fullsized":"2024-04-04T21:02:27.363811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:05.463818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:05.463818+0000","last_clean_scrub_stamp":"2024-04-04T21:02:05.463818+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:18:20.734360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345929,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":91133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":96543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899345930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":651264,"data_stored":488973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":96543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509450,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":96543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":96543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":96543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":663552,"data_stored":494383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-04T21:02:33.790 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-04T21:02:33.790 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-04T21:02:33.790 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-04T21:02:33.790 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-04T21:02:33.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.898+0000 7f62ab972640 10 monclient: get_monmap_and_config 2024-04-04T21:02:33.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.898+0000 7f62ab972640 10 monclient: build_initial_monmap 2024-04-04T21:02:33.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.898+0000 7f62ab972640 10 monclient: monmap: 2024-04-04T21:02:33.899 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:33.899 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:33.899 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:02:33.899492+0000 2024-04-04T21:02:33.899 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:02:33.899492+0000 2024-04-04T21:02:33.899 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:33.899 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:33.899 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:02:33.899 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:02:33.899 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:02:33.899 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:02:33.899 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:02:33.900 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:33.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.898+0000 7f62ab972640 1 Processor -- start 2024-04-04T21:02:33.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.899+0000 7f62ab972640 1 -- start start 2024-04-04T21:02:33.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.899+0000 7f62ab972640 10 monclient: init 2024-04-04T21:02:33.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.899+0000 7f62ab972640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:33.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.899+0000 7f62ab972640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-04-04T21:02:33.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.899+0000 7f62ab972640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a40b5200 0x7f62a413f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:33.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.899+0000 7f62ab972640 10 monclient(hunting): picked mon.noname-b con 0x7f62a40b5200 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:33.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.899+0000 7f62ab972640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62a40a9aa0 addr v1:172.21.15.203:6789/0 2024-04-04T21:02:33.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.899+0000 7f62ab972640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62a4146200 addr v1:172.21.15.106:6789/0 2024-04-04T21:02:33.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.899+0000 7f62ab972640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40a97e0 con 0x7f62a4146200 2024-04-04T21:02:33.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.899+0000 7f62ab972640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40ab420 con 0x7f62a40a9aa0 2024-04-04T21:02:33.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.899+0000 7f62ab972640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:33.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.899+0000 7f62ab972640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40b55e0 con 0x7f62a40b5200 2024-04-04T21:02:33.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.899+0000 7f62ab972640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:33.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.899+0000 7f62ab972640 10 monclient(hunting): authenticate will time out at 1403.167032s 2024-04-04T21:02:33.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.900+0000 7f62a9f0f640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f62a4146200 0x7f62a414ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:53440/0 (socket says 172.21.15.106:53440) 2024-04-04T21:02:33.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.900+0000 7f62a9f0f640 1 -- 172.21.15.106:0/2703108504 learned_addr learned my addr 172.21.15.106:0/2703108504 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:02:33.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.900+0000 7f62a970e640 1 --2- 172.21.15.106:0/2703108504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a40b5200 0x7f62a413f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:33.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.900+0000 7f62a970e640 10 monclient(hunting): get_auth_request con 0x7f62a40b5200 auth_method 0 2024-04-04T21:02:33.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.900+0000 7f62a970e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:33.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.900+0000 7f62a970e640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:33.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.900+0000 7f62a970e640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:33.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.901+0000 7f62a970e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:33.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.901+0000 7f62a970e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:33.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.901+0000 7f62a970e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:33.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.902+0000 7f629bfff640 1 -- 172.21.15.106:0/2703108504 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703619012 0 0) 0x7f62a40ab420 con 0x7f62a40a9aa0 2024-04-04T21:02:33.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.902+0000 7f629bfff640 10 monclient(hunting): _negotiate 2024-04-04T21:02:33.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.902+0000 7f629bfff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:33.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.902+0000 7f629bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.902+0000 7f629bfff640 10 monclient(hunting): my global_id is 4426 2024-04-04T21:02:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.902+0000 7f629bfff640 1 -- 172.21.15.106:0/2703108504 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6290003520 con 0x7f62a40a9aa0 2024-04-04T21:02:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.902+0000 7f629bfff640 1 -- 172.21.15.106:0/2703108504 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391485478 0 0) 0x7f62a40a97e0 con 0x7f62a4146200 2024-04-04T21:02:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.902+0000 7f629bfff640 10 monclient(hunting): _negotiate 2024-04-04T21:02:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.902+0000 7f629bfff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.902+0000 7f629bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.902+0000 7f629bfff640 10 monclient(hunting): my global_id is 4392 2024-04-04T21:02:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.902+0000 7f629bfff640 1 -- 172.21.15.106:0/2703108504 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a40ab420 con 0x7f62a4146200 2024-04-04T21:02:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.902+0000 7f62a970e640 10 monclient(hunting): handle_auth_done global_id 4370 payload 274 2024-04-04T21:02:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.903+0000 7f62a970e640 1 -- 172.21.15.106:0/2703108504 >> v1:172.21.15.203:6789/0 conn(0x7f62a40a9aa0 legacy=0x7f62a4145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.903+0000 7f62a970e640 1 -- 172.21.15.106:0/2703108504 >> v1:172.21.15.106:6789/0 conn(0x7f62a4146200 legacy=0x7f62a414ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.903+0000 7f62a970e640 10 monclient: _finish_hunting 0 2024-04-04T21:02:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.903+0000 7f62a970e640 1 monclient: found mon.noname-b 2024-04-04T21:02:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.903+0000 7f62a970e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:33.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.903+0000 7f62a970e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:02:33.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.903+0000 7f62a970e640 1 -- 172.21.15.106:0/2703108504 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a40ab700 con 0x7f62a40b5200 2024-04-04T21:02:33.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.904+0000 7f62a970e640 1 --2- 172.21.15.106:0/2703108504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a40b5200 0x7f62a413f9b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f62a00097d0 tx=0x7f62a002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1800a605c5256a02 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:33.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f629bfff640 1 -- 172.21.15.106:0/2703108504 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0035020 con 0x7f62a40b5200 2024-04-04T21:02:33.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f629bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:33.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f629bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:02:33.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f629bfff640 10 monclient: dump: 2024-04-04T21:02:33.909 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:33.909 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:33.909 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.909 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.909 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:33.909 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:33.909 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:33.909 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:33.909 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:33.910 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:33.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f629bfff640 10 monclient: _finish_auth 0 2024-04-04T21:02:33.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f629bfff640 10 monclient: _check_auth_tickets 2024-04-04T21:02:33.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f629bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:33.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f629bfff640 1 -- 172.21.15.106:0/2703108504 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a0031800 con 0x7f62a40b5200 2024-04-04T21:02:33.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f629bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:33.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f629bfff640 1 -- 172.21.15.106:0/2703108504 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0031bd0 con 0x7f62a40b5200 2024-04-04T21:02:33.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f629bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:33.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f629bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:02:33.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f629bfff640 10 monclient: dump: 2024-04-04T21:02:33.910 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:33.910 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:33.910 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.911 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.911 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:33.911 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:33.911 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:33.911 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:33.911 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:33.911 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:33.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f62ab972640 5 monclient: authenticate success, global_id 4370 2024-04-04T21:02:33.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f62ab972640 10 monclient: get_monmap_and_config success 2024-04-04T21:02:33.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.908+0000 7f62ab972640 4 set_mon_vals no callback set 2024-04-04T21:02:33.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.909+0000 7f62ab972640 10 monclient: shutdown 2024-04-04T21:02:33.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.909+0000 7f62ab972640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:33.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.909+0000 7f62ab972640 1 -- 172.21.15.106:0/2703108504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a40b5200 msgr2=0x7f62a413f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:33.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.909+0000 7f62ab972640 1 --2- 172.21.15.106:0/2703108504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a40b5200 0x7f62a413f9b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f62a00097d0 tx=0x7f62a002b8a0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.909+0000 7f62ab972640 1 -- 172.21.15.106:0/2703108504 shutdown_connections 2024-04-04T21:02:33.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.909+0000 7f62ab972640 1 --2- 172.21.15.106:0/2703108504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a40b5200 0x7f62a413f9b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.909+0000 7f62ab972640 1 -- 172.21.15.106:0/2703108504 >> 172.21.15.106:0/2703108504 conn(0x7f62a4066e40 msgr2=0x7f62a4067250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:33.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.909+0000 7f62ab972640 1 -- 172.21.15.106:0/2703108504 shutdown_connections 2024-04-04T21:02:33.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.909+0000 7f62ab972640 1 -- 172.21.15.106:0/2703108504 wait complete. 2024-04-04T21:02:33.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.910+0000 7f62ab972640 10 monclient: build_initial_monmap 2024-04-04T21:02:33.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.910+0000 7f62ab972640 10 monclient: monmap: 2024-04-04T21:02:33.916 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:02:33.916 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:02:33.916 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:02:33.916 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:02:33.916 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:02:33.916 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:33.916 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:02:33.916 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:02:33.916 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:02:33.916 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:33.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.910+0000 7f62ab972640 1 librados: starting msgr at 2024-04-04T21:02:33.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.910+0000 7f62ab972640 1 librados: starting objecter 2024-04-04T21:02:33.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.910+0000 7f62ab972640 1 Processor -- start 2024-04-04T21:02:33.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.910+0000 7f62ab972640 1 -- start start 2024-04-04T21:02:33.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.910+0000 7f62ab972640 1 librados: setting wanted keys 2024-04-04T21:02:33.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.910+0000 7f62ab972640 1 librados: calling monclient init 2024-04-04T21:02:33.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.910+0000 7f62ab972640 10 monclient: init 2024-04-04T21:02:33.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 10 monclient: _reopen_session rank -1 2024-04-04T21:02:33.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:02:33.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a40a9aa0 0x7f62a40ac0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:33.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 10 monclient(hunting): picked mon.noname-c con 0x7f62a40a9aa0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:02:33.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62a40b5200 0x7f62a41daee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:33.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 10 monclient(hunting): picked mon.noname-b con 0x7f62a40b5200 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:02:33.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a4146200 0x7f62a41db420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:33.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 10 monclient(hunting): picked mon.noname-a con 0x7f62a4146200 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:02:33.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:33.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a4140070 con 0x7f62a4146200 2024-04-04T21:02:33.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:33.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a413fef0 con 0x7f62a40b5200 2024-04-04T21:02:33.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 10 monclient(hunting): start opening mon connection 2024-04-04T21:02:33.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a41401f0 con 0x7f62a40a9aa0 2024-04-04T21:02:33.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 10 monclient(hunting): _renew_subs 2024-04-04T21:02:33.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62ab972640 10 monclient(hunting): authenticate will time out at 1403.178699s 2024-04-04T21:02:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62a970e640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a40a9aa0 0x7f62a40ac0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.911+0000 7f62a9f0f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a4146200 0x7f62a41db420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a8f0d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62a40b5200 0x7f62a41daee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a970e640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a40a9aa0 0x7f62a40ac0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:51884/0 (socket says 172.21.15.106:51884) 2024-04-04T21:02:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a970e640 1 -- 172.21.15.106:0/55384468 learned_addr learned my addr 172.21.15.106:0/55384468 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:02:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a970e640 10 monclient(hunting): get_auth_request con 0x7f62a40a9aa0 auth_method 0 2024-04-04T21:02:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a970e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a970e640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a970e640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a9f0f640 10 monclient(hunting): get_auth_request con 0x7f62a4146200 auth_method 0 2024-04-04T21:02:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a9f0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a9f0f640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a9f0f640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:33.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a8f0d640 10 monclient(hunting): get_auth_request con 0x7f62a40b5200 auth_method 0 2024-04-04T21:02:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a8f0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:02:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a8f0d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:02:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a8f0d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:02:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a970e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a970e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a970e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a8f0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a8f0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a8f0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a9f0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:02:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a9f0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:02:33.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.912+0000 7f62a9f0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:02:33.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.915+0000 7f62a9f0f640 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2024-04-04T21:02:33.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.915+0000 7f62a9f0f640 1 -- 172.21.15.106:0/55384468 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a40a9aa0 msgr2=0x7f62a40ac0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:33.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.915+0000 7f62a9f0f640 1 --2- 172.21.15.106:0/55384468 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a40a9aa0 0x7f62a40ac0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.915+0000 7f62a9f0f640 1 -- 172.21.15.106:0/55384468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62a40b5200 msgr2=0x7f62a41daee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:33.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.915+0000 7f62a9f0f640 1 --2- 172.21.15.106:0/55384468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62a40b5200 0x7f62a41daee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:33.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.915+0000 7f62a9f0f640 10 monclient: _finish_hunting 0 2024-04-04T21:02:33.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.915+0000 7f62a9f0f640 1 monclient: found mon.noname-a 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.915+0000 7f62a9f0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.915+0000 7f62a9f0f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.915+0000 7f62a9f0f640 1 -- 172.21.15.106:0/55384468 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a40adf00 con 0x7f62a4146200 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.915+0000 7f62a970e640 1 --2- 172.21.15.106:0/55384468 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a40a9aa0 0x7f62a40ac0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.917+0000 7f62a9f0f640 1 --2- 172.21.15.106:0/55384468 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a4146200 0x7f62a41db420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f629c00caa0 tx=0x7f629c00cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.917+0000 7f6299ffb640 10 client.?.objecter ms_handle_connect 0x7f62a4146200 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.917+0000 7f6299ffb640 10 client.?.objecter resend_mon_ops 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f6299ffb640 1 -- 172.21.15.106:0/55384468 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c017070 con 0x7f62a4146200 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f6299ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f6299ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f6299ffb640 10 monclient: dump: 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.923 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.924 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:33.924 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:33.924 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:33.924 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:33.924 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:33.924 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:33.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f6299ffb640 10 monclient: _finish_auth 0 2024-04-04T21:02:33.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f6299ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:02:33.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f6299ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:02:33.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f6299ffb640 1 -- 172.21.15.106:0/55384468 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629c00ab40 con 0x7f62a4146200 2024-04-04T21:02:33.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f6299ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:02:33.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f6299ffb640 1 -- 172.21.15.106:0/55384468 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c012400 con 0x7f62a4146200 2024-04-04T21:02:33.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f62ab972640 5 monclient: authenticate success, global_id 4395 2024-04-04T21:02:33.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f62ab972640 10 monclient: _renew_subs 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f62ab972640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f62ab972640 1 -- 172.21.15.106:0/55384468 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a40ae1e0 con 0x7f62a4146200 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f6299ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f6299ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f6299ffb640 10 monclient: dump: 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:02:33.925 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:02:33.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f629b7fe640 4 set_mon_vals no callback set 2024-04-04T21:02:33.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f62ab972640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:02:33.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f62ab972640 10 monclient: _renew_subs 2024-04-04T21:02:33.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f62ab972640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:33.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f62ab972640 1 -- 172.21.15.106:0/55384468 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a40ac610 con 0x7f62a4146200 2024-04-04T21:02:33.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.921+0000 7f62ab972640 1 librados: init done 2024-04-04T21:02:33.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.922+0000 7f62ab972640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:33.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.922+0000 7f62ab972640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:33.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.922+0000 7f62ab972640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:33.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.922+0000 7f62ab972640 1 -- 172.21.15.106:0/55384468 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a404f050 con 0x7f62a4146200 2024-04-04T21:02:33.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.940+0000 7f6299ffb640 1 -- 172.21.15.106:0/55384468 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f629c01a070 con 0x7f62a4146200 2024-04-04T21:02:33.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.940+0000 7f6299ffb640 1 --2- 172.21.15.106:0/55384468 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f628806ceb0 0x7f628806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:02:33.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.940+0000 7f62a970e640 1 --2- 172.21.15.106:0/55384468 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f628806ceb0 0x7f628806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:02:33.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.940+0000 7f62a970e640 10 monclient: get_auth_request con 0x7f628806ceb0 auth_method 0 2024-04-04T21:02:33.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.941+0000 7f62a970e640 1 --2- 172.21.15.106:0/55384468 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f628806ceb0 0x7f628806f370 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f62a002b8e0 tx=0x7f62a0008030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:02:33.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.941+0000 7f6299ffb640 10 client.4395.objecter ms_handle_connect 0x7f628806ceb0 2024-04-04T21:02:33.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.942+0000 7f6299ffb640 1 -- 172.21.15.106:0/55384468 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f629c056c40 con 0x7f62a4146200 2024-04-04T21:02:33.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.942+0000 7f6299ffb640 10 client.4395.objecter ms_dispatch 0x7f62a4000b90 osd_map(27..27 src has 1..27) v4 2024-04-04T21:02:33.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.942+0000 7f6299ffb640 3 client.4395.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:02:33.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.942+0000 7f6299ffb640 3 client.4395.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:02:33.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:33.942+0000 7f6299ffb640 20 client.4395.objecter dump_active .. 0 homeless 2024-04-04T21:02:34.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.179+0000 7f6299ffb640 1 -- 172.21.15.106:0/55384468 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f629c058ba0 con 0x7f62a4146200 2024-04-04T21:02:34.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.179+0000 7f6299ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:02:34.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.179+0000 7f6299ffb640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:02:34.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.396+0000 7f62ab972640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-04-04T21:02:34.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.396+0000 7f62ab972640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-04-04T21:02:34.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.396+0000 7f62ab972640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:02:34.397 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.396+0000 7f62ab972640 1 -- 172.21.15.106:0/55384468 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f62a4067dd0 con 0x7f62a4146200 2024-04-04T21:02:34.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.404+0000 7f6299ffb640 1 -- 172.21.15.106:0/55384468 <== mon.0 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f629c05cc40 con 0x7f62a4146200 2024-04-04T21:02:34.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.404+0000 7f6299ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-04-04T21:02:34.405 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.404+0000 7f6299ffb640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:02:34.405 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:02:34.405 INFO:teuthology.orchestra.run.smithi106.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-04T21:02:34.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 10 librados: watch_flush enter 2024-04-04T21:02:34.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 10 librados: watch_flush exit 2024-04-04T21:02:34.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 20 client.4395.objecter shutdown clearing up homeless session... 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 10 client.4395.objecter successfully canceled tick 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 1 -- 172.21.15.106:0/55384468 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f628806ceb0 msgr2=0x7f628806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 1 --2- 172.21.15.106:0/55384468 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f628806ceb0 0x7f628806f370 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f62a002b8e0 tx=0x7f62a0008030 comp rx=0 tx=0).stop 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 10 monclient: shutdown 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 1 -- 172.21.15.106:0/55384468 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a4146200 msgr2=0x7f62a41db420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 1 --2- 172.21.15.106:0/55384468 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a4146200 0x7f62a41db420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f629c00caa0 tx=0x7f629c00cea0 comp rx=0 tx=0).stop 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 1 -- 172.21.15.106:0/55384468 shutdown_connections 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 1 --2- 172.21.15.106:0/55384468 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f628806ceb0 0x7f628806f370 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 1 --2- 172.21.15.106:0/55384468 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a4146200 0x7f62a41db420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 1 --2- 172.21.15.106:0/55384468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62a40b5200 0x7f62a41daee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 1 --2- 172.21.15.106:0/55384468 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a40a9aa0 0x7f62a40ac0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 1 -- 172.21.15.106:0/55384468 >> 172.21.15.106:0/55384468 conn(0x7f62a4066e40 msgr2=0x7f62a40b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 1 -- 172.21.15.106:0/55384468 shutdown_connections 2024-04-04T21:02:34.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.406+0000 7f62ab972640 1 -- 172.21.15.106:0/55384468 wait complete. 2024-04-04T21:02:34.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:02:34.407+0000 7f62ab972640 1 librados: shutdown 2024-04-04T21:02:34.418 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-04T21:02:34.418 INFO:teuthology.run_tasks:Running task workunit... 2024-04-04T21:02:34.432 INFO:tasks.workunit:Pulling workunits from ref 66deef1598934c3b1ef2f044e380a7d00244cead 2024-04-04T21:02:34.432 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-04T21:02:34.433 DEBUG:teuthology.orchestra.run.smithi106:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-04T21:02:34.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-04T21:02:34.448 INFO:teuthology.orchestra.run.smithi106.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-04T21:02:34.448 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-04T21:02:34.503 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-04T21:02:34.503 DEBUG:teuthology.orchestra.run.smithi106:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-04T21:02:34.558 INFO:tasks.workunit:timeout=6h 2024-04-04T21:02:34.558 INFO:tasks.workunit:cleanup=True 2024-04-04T21:02:34.558 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 66deef1598934c3b1ef2f044e380a7d00244cead 2024-04-04T21:02:34.614 INFO:tasks.workunit.client.0.smithi106.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-04T21:02:38.088 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:01:29.605 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:38.091 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:01:29.608 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:38.130 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:01:29.647 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:38.137 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:29.620 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:38.139 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:29.622 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:38.140 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:01:29.657 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:38.142 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:01:29.659 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:38.178 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:29.661 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:38.191 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:01:29.706 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:58.083 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:01:49.599 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:58.132 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:01:49.648 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:58.145 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:01:49.662 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:58.181 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:49.663 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:58.182 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:49.665 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:58.198 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:49.680 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:58.202 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:01:49.719 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:58.204 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:01:49.721 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:02:58.218 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:01:49.735 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:18.186 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:02:09.703 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:18.189 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:02:09.706 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:18.208 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:02:09.725 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:18.230 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:02:09.747 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:18.241 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:09.724 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:18.242 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:09.725 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:18.256 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:09.738 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:18.260 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:02:09.777 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:18.285 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:02:09.801 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:32.653 INFO:tasks.workunit.client.0.smithi106.stderr:Note: switching to '66deef1598934c3b1ef2f044e380a7d00244cead'. 2024-04-04T21:03:32.653 INFO:tasks.workunit.client.0.smithi106.stderr: 2024-04-04T21:03:32.653 INFO:tasks.workunit.client.0.smithi106.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-04T21:03:32.653 INFO:tasks.workunit.client.0.smithi106.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-04T21:03:32.654 INFO:tasks.workunit.client.0.smithi106.stderr:state without impacting any branches by switching back to a branch. 2024-04-04T21:03:32.654 INFO:tasks.workunit.client.0.smithi106.stderr: 2024-04-04T21:03:32.654 INFO:tasks.workunit.client.0.smithi106.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-04T21:03:32.654 INFO:tasks.workunit.client.0.smithi106.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-04T21:03:32.654 INFO:tasks.workunit.client.0.smithi106.stderr: 2024-04-04T21:03:32.654 INFO:tasks.workunit.client.0.smithi106.stderr: git switch -c 2024-04-04T21:03:32.654 INFO:tasks.workunit.client.0.smithi106.stderr: 2024-04-04T21:03:32.654 INFO:tasks.workunit.client.0.smithi106.stderr:Or undo this operation with: 2024-04-04T21:03:32.654 INFO:tasks.workunit.client.0.smithi106.stderr: 2024-04-04T21:03:32.654 INFO:tasks.workunit.client.0.smithi106.stderr: git switch - 2024-04-04T21:03:32.654 INFO:tasks.workunit.client.0.smithi106.stderr: 2024-04-04T21:03:32.654 INFO:tasks.workunit.client.0.smithi106.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-04T21:03:32.654 INFO:tasks.workunit.client.0.smithi106.stderr: 2024-04-04T21:03:32.654 INFO:tasks.workunit.client.0.smithi106.stderr:HEAD is now at 66deef15989 test/ceph_test_rados: fix seeking over virtual, in-memory objects 2024-04-04T21:03:32.659 DEBUG:teuthology.orchestra.run.smithi106:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-04T21:03:32.727 INFO:tasks.workunit.client.0.smithi106.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-04T21:03:32.729 INFO:tasks.workunit.client.0.smithi106.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-04T21:03:32.729 INFO:tasks.workunit.client.0.smithi106.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-04T21:03:33.396 INFO:tasks.workunit.client.0.smithi106.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-04T21:03:33.465 INFO:tasks.workunit.client.0.smithi106.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-04T21:03:33.507 INFO:tasks.workunit.client.0.smithi106.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-04T21:03:33.508 INFO:tasks.workunit.client.0.smithi106.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-04T21:03:33.508 INFO:tasks.workunit.client.0.smithi106.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-04T21:03:33.548 INFO:tasks.workunit.client.0.smithi106.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-04T21:03:33.551 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-04T21:03:33.551 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-04T21:03:33.607 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-04-04T21:03:33.608 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-04-04T21:03:33.608 DEBUG:teuthology.orchestra.run.smithi106:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=66deef1598934c3b1ef2f044e380a7d00244cead TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-04-04T21:03:33.666 INFO:tasks.workunit.client.0.smithi106.stderr:+ parallel=1 2024-04-04T21:03:33.666 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' '' = --serial ']' 2024-04-04T21:03:33.666 INFO:tasks.workunit.client.0.smithi106.stderr:+ crimson=0 2024-04-04T21:03:33.667 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' '' = --crimson ']' 2024-04-04T21:03:33.667 INFO:tasks.workunit.client.0.smithi106.stderr:+ color= 2024-04-04T21:03:33.667 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -t 1 ']' 2024-04-04T21:03:33.667 INFO:tasks.workunit.client.0.smithi106.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-04-04T21:03:33.667 INFO:tasks.workunit.client.0.smithi106.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-04-04T21:03:33.667 INFO:tasks.workunit.client.0.smithi106.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-04-04T21:03:33.668 INFO:tasks.workunit.client.0.smithi106.stderr:+ declare -A pids 2024-04-04T21:03:33.668 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.668 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.668 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_aio 2024-04-04T21:03:33.668 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_aio' 2024-04-04T21:03:33.668 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_aio 2024-04-04T21:03:33.668 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.669 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_aio 2024-04-04T21:03:33.669 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41122 2024-04-04T21:03:33.669 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_aio on pid 41122' 2024-04-04T21:03:33.670 INFO:tasks.workunit.client.0.smithi106.stdout:test api_aio on pid 41122 2024-04-04T21:03:33.673 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41122 2024-04-04T21:03:33.673 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.674 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.674 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-04-04T21:03:33.674 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_aio_pp 2024-04-04T21:03:33.674 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_aio_pp' 2024-04-04T21:03:33.674 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_aio_pp 2024-04-04T21:03:33.674 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.674 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.674 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.674 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.674 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.674 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.674 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.675 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.675 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.675 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.675 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.675 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.675 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.675 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.675 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.675 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.675 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.675 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.675 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.676 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.676 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.676 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.676 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.676 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.676 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.676 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.676 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_aio_pp 2024-04-04T21:03:33.676 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41129 2024-04-04T21:03:33.677 INFO:tasks.workunit.client.0.smithi106.stdout:test api_aio_pp on pid 41129 2024-04-04T21:03:33.681 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_aio_pp on pid 41129' 2024-04-04T21:03:33.681 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41129 2024-04-04T21:03:33.681 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.681 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-04-04T21:03:33.681 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.681 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_io 2024-04-04T21:03:33.681 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_io' 2024-04-04T21:03:33.681 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_io 2024-04-04T21:03:33.681 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.681 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.681 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.681 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.682 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.682 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.682 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.682 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.682 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.682 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.682 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.682 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.682 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.682 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.682 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.682 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.682 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.682 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.683 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.684 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.684 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.684 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.684 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.684 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.684 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.684 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.684 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.684 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.684 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.684 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.684 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.684 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_io 2024-04-04T21:03:33.684 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41139 2024-04-04T21:03:33.685 INFO:tasks.workunit.client.0.smithi106.stdout:test api_io on pid 41139 2024-04-04T21:03:33.690 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_io on pid 41139' 2024-04-04T21:03:33.690 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41139 2024-04-04T21:03:33.690 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.691 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.691 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-04-04T21:03:33.691 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_io_pp 2024-04-04T21:03:33.691 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_io_pp' 2024-04-04T21:03:33.691 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.691 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_io_pp 2024-04-04T21:03:33.691 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.691 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.691 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.691 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.691 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.691 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.691 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.692 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.692 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.692 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.692 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.692 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.692 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.692 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.692 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.692 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.692 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.692 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.692 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.692 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.692 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.693 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.694 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.695 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.695 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.695 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.695 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_io_pp 2024-04-04T21:03:33.695 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41150 2024-04-04T21:03:33.695 INFO:tasks.workunit.client.0.smithi106.stdout:test api_io_pp on pid 41150 2024-04-04T21:03:33.700 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_io_pp on pid 41150' 2024-04-04T21:03:33.701 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41150 2024-04-04T21:03:33.701 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.701 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.701 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-04-04T21:03:33.701 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.701 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_asio 2024-04-04T21:03:33.701 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.701 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.701 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.701 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.701 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.701 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.701 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_asio' 2024-04-04T21:03:33.701 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_asio 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.702 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.703 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.704 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.705 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_asio 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41165 2024-04-04T21:03:33.706 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_asio on pid 41165' 2024-04-04T21:03:33.707 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-04-04T21:03:33.707 INFO:tasks.workunit.client.0.smithi106.stdout:test api_asio on pid 41165 2024-04-04T21:03:33.715 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41165 2024-04-04T21:03:33.715 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.715 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.715 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.715 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.715 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.715 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.715 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.715 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.715 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.715 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.716 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.716 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.716 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.716 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.716 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.716 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.716 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.716 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.716 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.716 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.716 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.716 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.716 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.716 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.717 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_list 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_list' 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41122/exe 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_list 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.718 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.719 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.720 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.721 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:33.722 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:33.723 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:33.723 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:33.723 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:33.723 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_list 2024-04-04T21:03:33.723 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41185 2024-04-04T21:03:33.723 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_list on pid 41185' 2024-04-04T21:03:33.723 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.723 INFO:tasks.workunit.client.0.smithi106.stdout:test api_list on pid 41185 2024-04-04T21:03:33.732 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.732 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41185 2024-04-04T21:03:33.732 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.732 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.732 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_aio.log 2024-04-04T21:03:33.733 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.734 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.734 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.734 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.734 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.734 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.734 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.734 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.734 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.734 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.734 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.734 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.734 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.734 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.734 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.735 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_aio: /' 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.736 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_lock 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_lock' 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41129/exe 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.737 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.738 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.738 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.738 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.738 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.738 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.738 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.738 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.738 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.738 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.738 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.738 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.738 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_lock 2024-04-04T21:03:33.738 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.738 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.739 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.739 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.739 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.739 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.739 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.739 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.739 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:33.739 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.739 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:33.739 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:33.739 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.739 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.739 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:33.740 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.741 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_lock 2024-04-04T21:03:33.741 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.741 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.741 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.741 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.741 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.741 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41206 2024-04-04T21:03:33.741 INFO:tasks.workunit.client.0.smithi106.stdout:test api_lock on pid 41206 2024-04-04T21:03:33.751 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_lock on pid 41206' 2024-04-04T21:03:33.751 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41206 2024-04-04T21:03:33.751 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.751 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.751 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.751 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.751 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.751 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.751 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.751 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-04-04T21:03:33.751 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.752 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.752 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.752 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_aio_pp: /' 2024-04-04T21:03:33.752 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.752 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.752 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.752 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.752 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_lock_pp 2024-04-04T21:03:33.752 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.752 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-04-04T21:03:33.752 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.752 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.752 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.752 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.753 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.753 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.753 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.753 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.753 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.753 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.753 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.753 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.753 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.753 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.753 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.753 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.754 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.754 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.754 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.754 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.754 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.754 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.754 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.754 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.754 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.754 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.754 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.754 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_lock_pp' 2024-04-04T21:03:33.754 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.754 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-04-04T21:03:33.755 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.755 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.755 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.755 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.755 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.755 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.755 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.755 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.755 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.755 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.755 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.755 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.755 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.755 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_lock_pp 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.756 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.757 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.758 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.758 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.758 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.758 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.758 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.758 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.758 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.758 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.758 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.758 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.758 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41139/exe 2024-04-04T21:03:33.758 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.758 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.762 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.762 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.762 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.762 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.762 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_lock_pp 2024-04-04T21:03:33.762 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.762 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.762 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.762 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.762 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41224 2024-04-04T21:03:33.762 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.762 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.762 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.763 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.763 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.763 INFO:tasks.workunit.client.0.smithi106.stdout:test api_lock_pp on pid 41224 2024-04-04T21:03:33.784 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_lock_pp on pid 41224' 2024-04-04T21:03:33.784 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.784 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41224 2024-04-04T21:03:33.784 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.784 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.784 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.784 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-04-04T21:03:33.784 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_misc 2024-04-04T21:03:33.784 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.785 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:33.785 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:33.785 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_misc' 2024-04-04T21:03:33.785 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:33.785 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:33.785 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.785 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.785 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.785 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.785 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.785 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.785 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.785 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.785 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.786 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.786 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.786 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:33.786 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.786 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:33.786 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:33.786 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.786 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.786 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.786 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.786 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.786 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.786 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.786 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.787 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.787 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.787 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.787 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.787 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.787 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.787 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.787 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.787 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.787 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.787 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.787 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.787 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.788 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.788 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.788 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.788 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.788 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.788 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.788 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.788 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.788 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:33.788 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:33.788 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:33.788 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.788 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.788 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_misc 2024-04-04T21:03:33.789 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.789 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.789 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_io.log 2024-04-04T21:03:33.789 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.789 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.789 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-04-04T21:03:33.789 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.789 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.789 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.789 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.789 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.789 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.789 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.789 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.790 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.790 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.790 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41150/exe 2024-04-04T21:03:33.790 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.790 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.790 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.790 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.790 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.790 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.790 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.790 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.790 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.790 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.790 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.791 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.791 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.791 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.791 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.791 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.791 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.791 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.791 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.791 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.791 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.791 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.791 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.791 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.792 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.792 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.792 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.792 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.792 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:33.792 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:33.792 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:33.792 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:33.792 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:33.792 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:33.792 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:33.792 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:33.793 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:33.793 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:33.793 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_io_pp: /' 2024-04-04T21:03:33.793 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.793 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.793 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.793 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.793 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.793 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.793 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.793 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.793 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.793 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.793 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.794 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.794 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.794 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.794 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.794 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.794 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.794 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-04-04T21:03:33.794 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.794 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.794 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.794 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.794 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.794 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.794 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.795 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.795 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.795 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.795 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.795 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.795 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.795 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.795 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.795 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.795 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.795 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.795 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.795 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.795 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.796 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.796 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.796 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-04-04T21:03:33.796 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.796 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.796 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.796 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.796 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.796 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.796 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.796 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.796 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.796 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.797 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.797 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.797 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.797 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.797 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.797 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.797 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.797 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.797 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.797 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.797 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.797 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.797 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.798 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.798 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.798 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.798 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.798 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.798 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.798 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.798 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.798 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.798 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.798 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.798 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.798 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.798 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.799 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.799 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.799 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.799 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.799 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.799 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.799 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.799 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.799 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.799 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.799 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.799 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.799 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.799 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.800 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.800 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.800 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.800 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.800 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.800 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.800 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.800 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.800 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41165/exe 2024-04-04T21:03:33.800 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.800 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.800 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.800 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.800 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.801 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.802 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.802 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41185/exe 2024-04-04T21:03:33.802 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.802 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:33.802 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:33.802 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:33.802 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.802 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:33.802 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:33.802 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.802 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:33.802 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:33.802 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.802 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.803 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_misc 2024-04-04T21:03:33.804 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.804 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41263 2024-04-04T21:03:33.804 INFO:tasks.workunit.client.0.smithi106.stdout:test api_misc on pid 41263 2024-04-04T21:03:33.807 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_misc on pid 41263' 2024-04-04T21:03:33.807 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41263 2024-04-04T21:03:33.807 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.807 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.807 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-04-04T21:03:33.807 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_asio.log 2024-04-04T21:03:33.807 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_misc_pp 2024-04-04T21:03:33.807 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:33.807 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_misc_pp' 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_misc_pp 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.808 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.809 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.809 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.809 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-04-04T21:03:33.809 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.809 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_io: /' 2024-04-04T21:03:33.809 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_misc_pp 2024-04-04T21:03:33.809 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41275 2024-04-04T21:03:33.809 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_misc_pp on pid 41275' 2024-04-04T21:03:33.809 INFO:tasks.workunit.client.0.smithi106.stdout:test api_misc_pp on pid 41275 2024-04-04T21:03:33.814 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41275 2024-04-04T21:03:33.815 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.815 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.815 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.815 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.815 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.815 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.815 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.815 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.815 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.815 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.815 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.815 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.815 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.815 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.816 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_tier_pp 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_tier_pp' 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_tier_pp 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.817 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.818 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.819 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_tier_pp 2024-04-04T21:03:33.819 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41284 2024-04-04T21:03:33.819 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_tier_pp on pid 41284' 2024-04-04T21:03:33.819 INFO:tasks.workunit.client.0.smithi106.stdout:test api_tier_pp on pid 41284 2024-04-04T21:03:33.832 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41284 2024-04-04T21:03:33.832 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.832 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.832 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-04-04T21:03:33.832 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_pool 2024-04-04T21:03:33.832 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_pool' 2024-04-04T21:03:33.832 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_asio: /' 2024-04-04T21:03:33.832 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.832 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.832 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.833 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.833 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.833 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.833 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.833 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.833 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.833 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.833 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.833 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.833 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.833 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.833 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.833 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.833 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_list.log 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.834 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_list: /' 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.835 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.836 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.837 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.837 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.837 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.837 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.837 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.837 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.837 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.837 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.837 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.837 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.837 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.837 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.837 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.837 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41224/exe 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:33.838 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_pool 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.839 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.840 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.841 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.842 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_pool 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.843 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.844 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41320 2024-04-04T21:03:33.844 INFO:tasks.workunit.client.0.smithi106.stdout:test api_pool on pid 41320 2024-04-04T21:03:33.852 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_pool on pid 41320' 2024-04-04T21:03:33.852 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41320 2024-04-04T21:03:33.852 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.852 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.852 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.852 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.852 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_snapshots 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.853 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_snapshots' 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_lock_pp: /' 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.854 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.855 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.856 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_snapshots 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41275/exe 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.857 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:33.858 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:33.859 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:33.859 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:33.859 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:33.859 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:33.859 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:33.859 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:33.859 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_snapshots 2024-04-04T21:03:33.859 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41357 2024-04-04T21:03:33.859 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_misc_pp: /' 2024-04-04T21:03:33.859 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_snapshots on pid 41357' 2024-04-04T21:03:33.859 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.859 INFO:tasks.workunit.client.0.smithi106.stdout:test api_snapshots on pid 41357 2024-04-04T21:03:33.867 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.867 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.867 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.867 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.867 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.867 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41357 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.868 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41206/exe 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_snapshots_pp 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_snapshots_pp' 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.869 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.870 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.870 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.870 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.870 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.870 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.870 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.870 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.870 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.870 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.870 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.870 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.870 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.870 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.871 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:33.871 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:33.871 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:33.871 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:33.871 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:33.871 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:33.871 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:33.871 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:33.871 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:33.871 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:33.871 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.871 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.871 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.871 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.872 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.873 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.873 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-04-04T21:03:33.873 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.873 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_lock.log 2024-04-04T21:03:33.873 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_snapshots_pp 2024-04-04T21:03:33.873 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_snapshots_pp 2024-04-04T21:03:33.873 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41384 2024-04-04T21:03:33.873 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_snapshots_pp on pid 41384' 2024-04-04T21:03:33.873 INFO:tasks.workunit.client.0.smithi106.stdout:test api_snapshots_pp on pid 41384 2024-04-04T21:03:33.885 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41384 2024-04-04T21:03:33.885 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.885 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.885 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.885 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.885 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.885 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.885 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_lock: /' 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.886 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.887 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.887 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.887 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.887 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.887 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.887 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.887 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.887 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.887 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.887 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.887 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.887 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.887 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.887 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.888 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.889 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_stat 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_stat' 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41263/exe 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41284/exe 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:33.890 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.891 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_stat 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.892 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_misc: /' 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_misc.log 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.893 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:33.894 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-04-04T21:03:33.895 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.895 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_stat 2024-04-04T21:03:33.895 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.895 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41407 2024-04-04T21:03:33.895 INFO:tasks.workunit.client.0.smithi106.stdout:test api_stat on pid 41407 2024-04-04T21:03:33.899 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_stat on pid 41407' 2024-04-04T21:03:33.899 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41407 2024-04-04T21:03:33.899 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.899 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.900 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.900 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.900 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.900 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.900 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.900 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.900 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.900 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.900 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.900 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.900 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.900 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.900 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.901 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.901 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.901 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.901 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-04-04T21:03:33.901 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.901 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_stat_pp 2024-04-04T21:03:33.901 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_stat_pp' 2024-04-04T21:03:33.901 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.901 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_stat_pp 2024-04-04T21:03:33.901 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.901 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.901 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.901 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.901 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.902 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.902 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.902 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.902 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.902 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.902 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.902 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.902 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.902 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.902 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.902 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.902 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.902 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.902 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.903 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.903 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.903 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.903 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.903 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.903 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.903 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.903 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-04-04T21:03:33.903 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_stat_pp 2024-04-04T21:03:33.903 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41425 2024-04-04T21:03:33.903 INFO:tasks.workunit.client.0.smithi106.stdout:test api_stat_pp on pid 41425 2024-04-04T21:03:33.909 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_stat_pp on pid 41425' 2024-04-04T21:03:33.909 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41425 2024-04-04T21:03:33.909 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.909 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.910 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.910 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.910 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.910 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.910 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.910 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-04-04T21:03:33.910 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_watch_notify 2024-04-04T21:03:33.910 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_watch_notify' 2024-04-04T21:03:33.910 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.910 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.910 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.910 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.910 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.911 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.911 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_tier_pp: /' 2024-04-04T21:03:33.911 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_watch_notify 2024-04-04T21:03:33.911 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.911 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.911 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.911 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.911 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.911 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.911 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.911 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.911 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.911 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.911 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.912 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.913 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.914 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.914 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.914 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.914 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.914 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.914 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.914 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.914 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.914 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.914 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.914 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.914 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.914 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.914 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.915 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.915 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.915 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.915 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.915 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.915 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_watch_notify 2024-04-04T21:03:33.915 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41441 2024-04-04T21:03:33.915 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_watch_notify on pid 41441' 2024-04-04T21:03:33.915 INFO:tasks.workunit.client.0.smithi106.stdout:test api_watch_notify on pid 41441 2024-04-04T21:03:33.916 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41441 2024-04-04T21:03:33.916 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.916 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.916 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_watch_notify_pp 2024-04-04T21:03:33.916 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_watch_notify_pp' 2024-04-04T21:03:33.916 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-04-04T21:03:33.917 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-04-04T21:03:33.917 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [----------] Global test environment set-up. 2024-04-04T21:03:33.917 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [----------] 12 tests from AsioRados 2024-04-04T21:03:33.917 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.792+0000 7f36f2653780 10 monclient: get_monmap_and_config 2024-04-04T21:03:33.917 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.792+0000 7f36f2653780 10 monclient: build_initial_monmap 2024-04-04T21:03:33.917 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.792+0000 7f36f2653780 10 monclient: monmap: 2024-04-04T21:03:33.917 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: epoch 0 2024-04-04T21:03:33.917 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:03:33.917 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: last_changed 0.000000 2024-04-04T21:03:33.917 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: created 0.000000 2024-04-04T21:03:33.917 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: min_mon_release 0 (unknown) 2024-04-04T21:03:33.917 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: election_strategy: 1 2024-04-04T21:03:33.917 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:03:33.918 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:03:33.918 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:03:33.918 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.918 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.793+0000 7f36f2653780 1 Processor -- start 2024-04-04T21:03:33.918 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.793+0000 7f36f2653780 1 -- start start 2024-04-04T21:03:33.918 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.793+0000 7f36f2653780 10 monclient: init 2024-04-04T21:03:33.918 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 10 monclient: _reopen_session rank -1 2024-04-04T21:03:33.918 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 10 monclient: _add_conns ranks=[1,2,0] 2024-04-04T21:03:33.918 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b3e652270 0x559b3e7201d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:03:33.918 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 10 monclient(hunting): picked mon.noname-b con 0x559b3e652270 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:03:33.918 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559b3e720b50 0x559b3e71a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:03:33.918 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 10 monclient(hunting): picked mon.noname-c con 0x559b3e720b50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:03:33.918 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559b3e72a7e0 0x559b3e695aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:03:33.918 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 10 monclient(hunting): picked mon.noname-a con 0x559b3e72a7e0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:03:33.919 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 10 monclient(hunting): start opening mon connection 2024-04-04T21:03:33.919 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3e68e0a0 con 0x559b3e72a7e0 2024-04-04T21:03:33.919 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 10 monclient(hunting): start opening mon connection 2024-04-04T21:03:33.919 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3e605b20 con 0x559b3e652270 2024-04-04T21:03:33.919 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 10 monclient(hunting): start opening mon connection 2024-04-04T21:03:33.919 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x559b3e68ef50 con 0x559b3e720b50 2024-04-04T21:03:33.919 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 10 monclient(hunting): _renew_subs 2024-04-04T21:03:33.919 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.794+0000 7f36f2653780 10 monclient(hunting): authenticate will time out at 1463.061485s 2024-04-04T21:03:33.923 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.923 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.923 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.923 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.923 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.924 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.925 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.926 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.927 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.927 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.927 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.927 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.795+0000 7f36f0954640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559b3e72a7e0 0x559b3e695aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:03:33.927 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.795+0000 7f36f0954640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559b3e72a7e0 0x559b3e695aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:46090/0 (socket says 172.21.15.106:46090) 2024-04-04T21:03:33.927 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.795+0000 7f36f0954640 1 -- 172.21.15.106:0/3102479791 learned_addr learned my addr 172.21.15.106:0/3102479791 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:03:33.927 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.795+0000 7f36f0954640 10 monclient(hunting): get_auth_request con 0x559b3e72a7e0 auth_method 0 2024-04-04T21:03:33.927 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.795+0000 7f36f0954640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:03:33.927 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.795+0000 7f36f0954640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:03:33.927 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.795+0000 7f36f0954640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:03:33.927 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.797+0000 7f36f0954640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:03:33.927 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.797+0000 7f36f0954640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.797+0000 7f36f0954640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.798+0000 7f36ef952640 1 --2- 172.21.15.106:0/3102479791 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559b3e720b50 0x559b3e71a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.798+0000 7f36ef952640 10 monclient(hunting): get_auth_request con 0x559b3e720b50 auth_method 0 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.798+0000 7f36ef952640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.798+0000 7f36ef952640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.798+0000 7f36ef952640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.799+0000 7f36ef952640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.799+0000 7f36ef952640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.799+0000 7f36ef952640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.800+0000 7f36f0153640 1 --2- 172.21.15.106:0/3102479791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b3e652270 0x559b3e7201d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.803+0000 7f36f0153640 10 monclient(hunting): get_auth_request con 0x559b3e652270 auth_method 0 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.803+0000 7f36f0153640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.803+0000 7f36f0153640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.803+0000 7f36f0153640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:03:33.928 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.803+0000 7f36ef952640 10 monclient(hunting): handle_auth_done global_id 4388 payload 274 2024-04-04T21:03:33.942 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.942 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.942 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.942 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.942 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.942 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.943 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.944 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.945 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.946 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41320/exe 2024-04-04T21:03:33.946 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.946 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.946 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.946 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.946 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.946 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.946 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.946 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.946 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.946 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.946 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.946 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.946 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.947 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.948 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_watch_notify_pp 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.949 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:33.950 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:33.950 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.950 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:33.950 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:33.950 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:33.950 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:33.950 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.950 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.950 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:33.950 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.950 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.950 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:33.950 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:33.950 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_pool: /' 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.951 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_pool.log 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.952 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:33.953 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:33.953 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.953 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41407/exe 2024-04-04T21:03:33.953 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.953 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_watch_notify_pp 2024-04-04T21:03:33.953 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41534 2024-04-04T21:03:33.953 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_watch_notify_pp on pid 41534' 2024-04-04T21:03:33.953 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-04-04T21:03:33.953 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.803+0000 7f36ef952640 1 -- 172.21.15.106:0/3102479791 >> [v2:172.test api_watch_notify_pp on pid 41534 2024-04-04T21:03:33.968 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41534 2024-04-04T21:03:33.968 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.968 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.969 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.969 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.969 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.969 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.969 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.969 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:33.969 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:33.969 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.969 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.969 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:33.969 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:33.969 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.969 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.969 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.970 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.971 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.971 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.971 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.971 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.971 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.971 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.971 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:33.971 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:33.971 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.971 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.971 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.971 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.971 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.971 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.972 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.972 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.972 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-04-04T21:03:33.972 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.972 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.972 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.972 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.972 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.972 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.972 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.972 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.972 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.972 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.972 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.973 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.974 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_cmd 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_cmd' 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.975 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.976 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41441/exe 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_stat.log 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_stat: /' 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:33.977 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.978 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:33.978 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:33.978 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.978 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:33.978 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.978 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:33.978 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.978 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:33.978 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.978 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:33.978 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:33.978 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.978 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.978 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.979 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_cmd 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41357/exe 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:33.980 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:33.981 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:33.981 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:33.981 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:33.981 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-04-04T21:03:33.981 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-04-04T21:03:33.981 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:33.981 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_cmd 2024-04-04T21:03:33.981 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41569 2024-04-04T21:03:33.981 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_cmd on pid 41569' 2024-04-04T21:03:33.981 INFO:tasks.workunit.client.0.smithi106.stdout:test api_cmd on pid 41569 2024-04-04T21:03:33.992 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41569 2024-04-04T21:03:33.992 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:33.992 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:33.992 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-04-04T21:03:33.992 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.993 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_watch_notify: /' 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.994 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.995 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_cmd_pp 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_cmd_pp' 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:33.996 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_cmd_pp 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:33.997 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:33.998 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:33.999 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41425/exe 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.000 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_stat_pp: /' 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_cmd_pp 2024-04-04T21:03:34.001 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41602 2024-04-04T21:03:34.002 INFO:tasks.workunit.client.0.smithi106.stdout:test api_cmd_pp on pid 41602 2024-04-04T21:03:34.009 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_cmd_pp on pid 41602' 2024-04-04T21:03:34.010 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41602 2024-04-04T21:03:34.010 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:34.010 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.010 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.010 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.010 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.010 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.010 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.010 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.010 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.010 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.010 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.010 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.010 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.011 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.012 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.012 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.012 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.012 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.012 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.012 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.012 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-04-04T21:03:34.012 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.012 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-04-04T21:03:34.012 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.012 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.012 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.012 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.012 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.013 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_snapshots: /' 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41384/exe 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_service 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_service' 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.014 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.015 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.016 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.016 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.016 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.016 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.016 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.016 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.016 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.016 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.016 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.016 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-04-04T21:03:34.016 INFO:tasks.workunit.client.0.smithi106.stdout:21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b3e652270 msgr2=0x559b3e7201d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:03:34.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.803+0000 7f36ef952640 1 --2- 172.21.15.106:0/3102479791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b3e652270 0x559b3e7201d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:03:34.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.803+0000 7f36ef952640 1 -- 172.21.15.106:0/3102479791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559b3e72a7e0 msgr2=0x559b3e695aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:03:34.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.803+0000 7f36ef952640 1 --2- 172.21.15.106:0/3102479791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559b3e72a7e0 0x559b3e695aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:03:34.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.803+0000 7f36ef952640 10 monclient: _finish_hunting 0 2024-04-04T21:03:34.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.803+0000 7f36ef952640 1 monclient: found mon.noname-c 2024-04-04T21:03:34.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.803+0000 7f36ef952640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:03:34.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.803+0000 7f36ef952640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:03:34.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.803+0000 7f36ef952640 1 -- 172.21.15.106:0/3102479791 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b3e71fd20 con 0x559b3e720b50 2024-04-04T21:03:34.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.859+0000 7f36ef952640 1 --2- 172.21.15.106:0/3102479791 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559b3e720b50 0x559b3e71a390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f36e40097a0 tx=0x7f36e4009890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab5519001266f6ef server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:03:34.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36ee950640 1 -- 172.21.15.106:0/3102479791 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4019070 con 0x559b3e720b50 2024-04-04T21:03:34.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36ee950640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:03:34.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36ee950640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:03:34.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36ee950640 10 monclient: dump: 2024-04-04T21:03:34.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: epoch 1 2024-04-04T21:03:34.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:03:34.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:03:34.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:03:34.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: min_mon_release 19 (squid) 2024-04-04T21:03:34.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: election_strategy: 1 2024-04-04T21:03:34.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:03:34.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:03:34.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:03:34.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36ee950640 10 monclient: _finish_auth 0 2024-04-04T21:03:34.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36ee950640 10 monclient: _check_auth_tickets 2024-04-04T21:03:34.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36ee950640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:34.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36ee950640 1 -- 172.21.15.106:0/3102479791 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e40086d0 con 0x559b3e720b50 2024-04-04T21:03:34.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36ee950640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:03:34.019 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.019 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.020 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.020 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.020 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.020 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_service 2024-04-04T21:03:34.020 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_service 2024-04-04T21:03:34.020 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-04-04T21:03:34.020 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41643 2024-04-04T21:03:34.020 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36ee950640 1 -- 172.21.15.106:0/3102479791 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (securtest api_service on pid 41643 2024-04-04T21:03:34.021 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_service on pid 41643' 2024-04-04T21:03:34.021 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41643 2024-04-04T21:03:34.021 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:34.021 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.021 INFO:tasks.workunit.client.0.smithi106.stdout:e 0 0 0) 0x7f36e4008aa0 con 0x559b3e720b50 2024-04-04T21:03:34.021 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36ee950640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:03:34.021 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36ee950640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:03:34.021 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36ee950640 10 monclient: dump: 2024-04-04T21:03:34.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: epoch 1 2024-04-04T21:03:34.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:03:34.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:03:34.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:03:34.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: min_mon_release 19 (squid) 2024-04-04T21:03:34.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: election_strategy: 1 2024-04-04T21:03:34.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:03:34.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:03:34.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:03:34.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36f2653780 5 monclient: authenticate success, global_id 4388 2024-04-04T21:03:34.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36f2653780 10 monclient: get_monmap_and_config success 2024-04-04T21:03:34.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.874+0000 7f36f2653780 4 set_mon_vals no callback set 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.876+0000 7f36f2653780 10 monclient: shutdown 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.876+0000 7f36f2653780 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.876+0000 7f36f2653780 1 -- 172.21.15.106:0/3102479791 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559b3e720b50 msgr2=0x559b3e71a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.876+0000 7f36f2653780 1 --2- 172.21.15.106:0/3102479791 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559b3e720b50 0x559b3e71a390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f36e40097a0 tx=0x7f36e4009890 comp rx=0 tx=0).stop 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.877+0000 7f36f2653780 1 -- 172.21.15.106:0/3102479791 shutdown_connections 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.877+0000 7f36f2653780 1 --2- 172.21.15.106:0/3102479791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559b3e72a7e0 0x559b3e695aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.877+0000 7f36f2653780 1 --2- 172.21.15.106:0/3102479791 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559b3e720b50 0x559b3e71a390 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.877+0000 7f36f2653780 1 --2- 172.21.15.106:0/3102479791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b3e652270 0x559b3e7201d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.877+0000 7f36f2653780 1 -- 172.21.15.106:0/3102479791 >> 172.21.15.106:0/3102479791 conn(0x559b3e697930 msgr2=0x559b3e71eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.877+0000 7f36f2653780 1 -- 172.21.15.106:0/3102479791 shutdown_connections 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.877+0000 7f36f2653780 1 -- 172.21.15.106:0/3102479791 wait complete. 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.877+0000 7f36f2653780 10 monclient: build_initial_monmap 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.877+0000 7f36f2653780 10 monclient: monmap: 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: epoch 0 2024-04-04T21:03:34.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:03:34.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: last_changed 0.000000 2024-04-04T21:03:34.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: created 0.000000 2024-04-04T21:03:34.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: min_mon_release 0 (unknown) 2024-04-04T21:03:34.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: election_strategy: 1 2024-04-04T21:03:34.027 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_service_pp 2024-04-04T21:03:34.027 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-04-04T21:03:34.028 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_service_pp' 2024-04-04T21:03:34.028 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_service_pp 2024-04-04T21:03:34.028 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.028 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-04-04T21:03:34.028 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.028 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.028 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.028 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.028 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.028 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.028 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.028 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.028 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.028 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.029 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.030 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.030 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.030 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_service_pp 2024-04-04T21:03:34.030 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.030 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.030 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.030 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41662 2024-04-04T21:03:34.030 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.030 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_service_pp on pid 41662' 2024-04-04T21:03:34.030 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-atest api_service_pp on pid 41662 2024-04-04T21:03:34.047 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.047 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.047 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41662 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-04-04T21:03:34.048 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.049 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.049 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.049 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.049 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.049 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.049 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.049 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.049 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.049 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.049 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.049 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.049 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.049 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.050 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.050 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.050 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.050 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.050 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.050 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.050 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.050 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.050 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.050 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.050 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.050 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.050 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.050 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41569/exe 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.051 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.052 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.053 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.053 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.053 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.053 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.053 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.053 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.053 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.053 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.053 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.053 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.053 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.053 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.053 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.053 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.054 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.054 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.054 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.054 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.054 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.054 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.054 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.054 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.054 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.054 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.054 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.054 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.054 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.054 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.055 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_cmd.log 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41534/exe 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_c_write_operations 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.056 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_c_write_operations' 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_c_write_operations 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.057 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.058 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-04-04T21:03:34.059 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_cmd: /' 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.060 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.061 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.061 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.061 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.061 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.061 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.061 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.061 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.061 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.061 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.061 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.061 INFO:tasks.workunit.client.0.smithi106.stdout: 2024-04-04T21:03:34.062 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:03:34.062 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:03:34.062 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.062 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.877+0000 7f36f2653780 1 librados: starting msgr at 2024-04-04T21:03:34.062 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.877+0000 7f36f2653780 1 librados: starting objecter 2024-04-04T21:03:34.062 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.878+0000 7f36f2653780 1 Processor -- start 2024-04-04T21:03:34.062 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.878+0000 7f36f2653780 1 -- start start 2024-04-04T21:03:34.062 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.878+0000 7f36f2653780 1 librados: setting wanted keys 2024-04-04T21:03:34.062 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.878+0000 7f36f2653780 1 librados: calling monclient init 2024-04-04T21:03:34.062 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.878+0000 7f36f2653780 10 monclient: init 2024-04-04T21:03:34.062 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.878+0000 7f36f2653780 10 monclient: _reopen_session rank -1 2024-04-04T21:03:34.062 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.878+0000 7f36f2653780 10 monclient: _add_conns ranks=[2,0,1] 2024-04-04T21:03:34.063 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.878+0000 7f36f2653780 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559b3e652270 0x559b3e7b6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:03:34.063 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.878+0000 7f36f2653780 10 monclient(hunting): picked mon.noname-c con 0x559b3e652270 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:03:34.063 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.878+0000 7f36f2653780 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559b3e720b50 0x559b3e718e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:03:34.063 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.879+0000 7f36f2653780 10 monclient(hunting): picked mon.noname-a con 0x559b3e720b50 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:03:34.063 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.879+0000 7f36f2653780 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b3e72a7e0 0x559b3e7193b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:03:34.063 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.879+0000 7f36f2653780 10 monclient(hunting): picked mon.noname-b con 0x559b3e72a7e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:03:34.063 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.879+0000 7f36f2653780 10 monclient(hunting): start opening mon connection 2024-04-04T21:03:34.063 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.879+0000 7f36f2653780 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3e7ba660 con 0x559b3e720b50 2024-04-04T21:03:34.063 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.879+0000 7f36f2653780 10 monclient(hunting): start opening mon connection 2024-04-04T21:03:34.063 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.879+0000 7f36f2653780 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b3e71ea50 con 0x559b3e72a7e0 2024-04-04T21:03:34.063 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.879+0000 7f36f2653780 10 monclient(hunting): start opening mon connection 2024-04-04T21:03:34.063 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.879+0000 7f36f2653780 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x559b3e693c60 con 0x559b3e652270 2024-04-04T21:03:34.063 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.879+0000 7f36f2653780 10 monclient(hunting): _renew_subs 2024-04-04T21:03:34.063 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.879+0000 7f36f2653780 10 monclient(hunting): authenticate will time out at 1463.146222s 2024-04-04T21:03:34.064 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.907+0000 7f36ef952640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559b3e720b50 0x559b3e718e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:03:34.066 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.066 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.066 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.066 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.066 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.066 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.067 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.067 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.067 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.067 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.067 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.067 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.067 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.067 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.067 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.067 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.067 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.067 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.067 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.067 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.068 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.068 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.068 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_c_write_operations 2024-04-04T21:03:34.068 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41724 2024-04-04T21:03:34.068 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_c_write_operations on pid 41724' 2024-04-04T21:03:34.068 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.910+0000 7f36f0954640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b3e72a7e0 0x559b3e7193b0 unknown :-1 s=BANNER_CONNECTING pgs=0 test api_c_write_operations on pid 41724 2024-04-04T21:03:34.076 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41724 2024-04-04T21:03:34.076 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:34.076 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.076 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.076 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.076 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-04-04T21:03:34.076 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.076 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.076 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.076 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_c_read_operations 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.077 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_c_read_operations' 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41602/exe 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.078 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.079 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.080 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.081 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.082 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.082 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.082 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.082 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.082 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.082 INFO:tasks.workunit.client.0.smithi106.stdout:cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:03:34.082 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.912+0000 7f36f0954640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b3e72a7e0 0x559b3e7193b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:51746/0 (socket says 172.21.15.106:51746) 2024-04-04T21:03:34.082 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.912+0000 7f36f0954640 1 -- 172.21.15.106:0/1606434848 learned_addr learned my addr 172.21.15.106:0/1606434848 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:03:34.082 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.912+0000 7f36f0954640 10 monclient(hunting): get_auth_request con 0x559b3e72a7e0 auth_method 0 2024-04-04T21:03:34.082 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.912+0000 7f36f0954640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:03:34.082 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.912+0000 7f36f0954640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:03:34.083 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.912+0000 7f36f0954640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:03:34.083 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.919+0000 7f36f0153640 1 --2- 172.21.15.106:0/1606434848 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559b3e652270 0x559b3e7b6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:03:34.083 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.919+0000 7f36f0153640 10 monclient(hunting): get_auth_request con 0x559b3e652270 auth_method 0 2024-04-04T21:03:34.083 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.919+0000 7f36f0153640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:03:34.083 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.919+0000 7f36f0153640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:03:34.083 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.919+0000 7f36f0153640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:03:34.083 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.921+0000 7f36f0153640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:03:34.083 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.921+0000 7f36f0153640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:03:34.083 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.921+0000 7f36f0153640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:03:34.083 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.926+0000 7f36f0954640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:03:34.083 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.926+0000 7f36f0954640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:03:34.083 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.926+0000 7f36f0954640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:03:34.083 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.934+0000 7f36f0954640 10 monclient(hunting): handle_auth_done global_id 4474 payload 824 2024-04-04T21:03:34.083 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.934+0000 7f36f0954640 1 -- 172.21.15.106:0/1606434848 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559b3e652270 msgr2=0x559b3e7b6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:03:34.084 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.934+0000 7f36f0954640 1 --2- 172.21.15.106:0/1606434848 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559b3e652270 0x559b3e7b6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:03:34.084 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.934+0000 7f36f0954640 1 -- 172.21.15.106:0/1606434848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559b3e720b50 msgr2=0x559b3e718e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:03:34.084 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.934+0000 7f36f0954640 1 --2- 172.21.15.106:0/1606434848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559b3e720b50 0x559b3e718e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:03:34.084 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_c_read_operations 2024-04-04T21:03:34.084 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.085 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_c_read_operations 2024-04-04T21:03:34.085 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-04-04T21:03:34.085 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41763 2024-04-04T21:03:34.085 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_c_read_operations on pid 41763' 2024-04-04T21:03:34.085 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.934+0000 7f36f0954640 10 monclitest api_c_read_operations on pid 41763 2024-04-04T21:03:34.088 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41763 2024-04-04T21:03:34.088 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:34.088 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.088 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.088 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.088 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-04-04T21:03:34.088 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.088 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s api_cls_remote_reads 2024-04-04T21:03:34.088 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.088 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.088 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' api_cls_remote_reads' 2024-04-04T21:03:34.088 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.088 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo api_cls_remote_reads 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.089 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.090 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.090 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.090 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.090 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.090 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.090 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.090 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.090 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.090 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=api_cls_remote_reads 2024-04-04T21:03:34.090 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.090 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41770 2024-04-04T21:03:34.090 INFO:tasks.workunit.client.0.smithi106.stdout:test api_cls_remote_reads on pid 41770 2024-04-04T21:03:34.103 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test api_cls_remote_reads on pid 41770' 2024-04-04T21:03:34.103 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41770 2024-04-04T21:03:34.103 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:34.103 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.104 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.104 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.104 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.104 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.104 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.104 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-04-04T21:03:34.104 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41643/exe 2024-04-04T21:03:34.104 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.104 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.104 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.104 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.104 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.104 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.104 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.105 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s list_parallel 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.106 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' list_parallel' 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.107 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_service: /' 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.108 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.109 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_service.log 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.110 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.111 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.111 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.111 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.111 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.111 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.111 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.111 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.111 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.111 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.111 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.111 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.111 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.111 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.111 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.112 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.112 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.112 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.112 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.112 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.112 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.112 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo list_parallel 2024-04-04T21:03:34.112 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.112 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.112 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.112 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.112 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.112 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.112 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.113 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.114 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=list_parallel 2024-04-04T21:03:34.115 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41813 2024-04-04T21:03:34.116 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.116 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.116 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test list_parallel on pid 41813' 2024-04-04T21:03:34.116 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.116 INFO:tasks.workunit.client.0.smithi106.stdout:test list_parallel on pid 41813 2024-04-04T21:03:34.123 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41813 2024-04-04T21:03:34.123 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:34.123 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.123 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.123 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.123 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.123 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.123 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.123 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.123 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.123 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.123 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.123 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.123 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.124 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41662/exe 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.125 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.126 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s open_pools_parallel 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' open_pools_parallel' 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo open_pools_parallel 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.127 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.128 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.128 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-04-04T21:03:34.128 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_service_pp: /' 2024-04-04T21:03:34.128 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-04-04T21:03:34.128 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=open_pools_parallel 2024-04-04T21:03:34.128 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41830 2024-04-04T21:03:34.128 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-04-04T21:03:34.128 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.128 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test open_pools_parallel on pid 41830' 2024-04-04T21:03:34.128 INFO:tasks.workunit.client.0.smithi106.stdout:test open_pools_parallel on pid 41830 2024-04-04T21:03:34.143 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41830 2024-04-04T21:03:34.143 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-04T21:03:34.143 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.143 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.143 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.143 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.144 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.144 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.144 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.144 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.144 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.144 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.144 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.144 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.144 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.144 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.144 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.144 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.144 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.145 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-04-04T21:03:34.146 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s delete_pools_parallel 2024-04-04T21:03:34.146 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' delete_pools_parallel' 2024-04-04T21:03:34.146 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.146 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.146 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.146 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.146 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.146 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.146 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.146 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.146 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.146 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.146 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.147 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.147 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.147 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.147 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.147 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.147 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.147 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.147 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.147 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.147 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.147 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.147 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.148 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.148 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.148 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.148 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.148 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.148 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.148 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.148 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.148 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.148 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.148 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.148 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.148 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.149 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.149 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.149 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.149 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.149 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.149 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.149 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.149 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.149 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.149 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.149 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.149 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.149 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.149 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.150 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.150 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.150 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.150 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.150 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.150 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.150 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.150 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.150 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.150 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.150 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.150 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.150 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.151 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.152 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41770/exe 2024-04-04T21:03:34.153 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.153 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.153 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.153 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.153 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.153 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.153 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.153 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.153 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.153 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.153 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.153 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.153 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.154 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.154 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.154 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.154 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.154 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.154 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.154 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.154 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.154 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.154 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.154 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.154 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.154 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.155 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.155 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.155 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.155 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.155 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.155 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.155 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.155 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.155 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.155 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.155 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.155 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.155 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.156 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.156 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.156 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.156 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.156 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.156 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.156 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.156 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.156 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.156 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.156 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.156 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.156 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.156 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo delete_pools_parallel 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.157 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.158 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.158 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.158 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.158 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41763/exe 2024-04-04T21:03:34.158 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.158 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.158 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.158 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.158 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=delete_pools_parallel 2024-04-04T21:03:34.158 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.158 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.158 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.158 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.158 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41863 2024-04-04T21:03:34.159 INFO:tasks.workunit.client.0.smithi106.stdout:test delete_pools_parallel on pid 41863 2024-04-04T21:03:34.173 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test delete_pools_parallel on pid 41863' 2024-04-04T21:03:34.173 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41863 2024-04-04T21:03:34.173 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.173 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-04-04T21:03:34.173 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-04T21:03:34.173 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.173 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.173 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.173 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.174 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.174 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.174 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.174 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.174 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.174 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.174 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s cls 2024-04-04T21:03:34.174 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.174 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.174 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' cls' 2024-04-04T21:03:34.174 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.174 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-04-04T21:03:34.174 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.174 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.175 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.176 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41724/exe 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo cls 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.177 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-04-04T21:03:34.178 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.179 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.180 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.181 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.182 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.182 INFO:tasks.workunit.client.0.smithi106.stdout:ent: _finish_hunting 0 2024-04-04T21:03:34.182 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.934+0000 7f36f0954640 1 monclient: found mon.noname-b 2024-04-04T21:03:34.182 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.934+0000 7f36f0954640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:03:34.182 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.934+0000 7f36f0954640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:03:34.182 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.934+0000 7f36f0954640 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b3e730cf0 con 0x559b3e72a7e0 2024-04-04T21:03:34.182 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.951+0000 7f36f0954640 1 --2- 172.21.15.106:0/1606434848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b3e72a7e0 0x559b3e7193b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f36e800b440 tx=0x7f36e800b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:03:34.182 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.951+0000 7f36ed94e640 10 client.?.objecter ms_handle_connect 0x559b3e72a7e0 2024-04-04T21:03:34.182 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.951+0000 7f36ed94e640 10 client.?.objecter resend_mon_ops 2024-04-04T21:03:34.182 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36ed94e640 1 -- 172.21.15.106:0/1606434848 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e800f580 con 0x559b3e72a7e0 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36ed94e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36ed94e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36ed94e640 10 monclient: dump: 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: epoch 1 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: min_mon_release 19 (squid) 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: election_strategy: 1 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36ed94e640 10 monclient: _finish_auth 0 2024-04-04T21:03:34.183 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36ed94e640 10 monclient: _check_auth_tickets 2024-04-04T21:03:34.184 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36ed94e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:34.184 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36ed94e640 1 -- 172.21.15.106:0/1606434848 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e800f740 con 0x559b3e72a7e0 2024-04-04T21:03:34.184 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36ed94e640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:03:34.184 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36ed94e640 1 -- 172.21.15.106:0/1606434848 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e800fb30 con 0x559b3e72a7e0 2024-04-04T21:03:34.184 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36ed94e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:03:34.184 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36ed94e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:03:34.184 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36ed94e640 10 monclient: dump: 2024-04-04T21:03:34.184 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: epoch 1 2024-04-04T21:03:34.184 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:03:34.184 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:03:34.184 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:03:34.184 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: min_mon_release 19 (squid) 2024-04-04T21:03:34.184 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: election_strategy: 1 2024-04-04T21:03:34.184 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:03:34.186 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41813/exe 2024-04-04T21:03:34.186 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=cls 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41910 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test cls on pid 41910' 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.187 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.188 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.188 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.188 INFO:tasks.workunit.client.0.smithi106.stdout: test cls on pid 41910 2024-04-04T21:03:34.196 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.196 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.196 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.196 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.196 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.196 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41910 2024-04-04T21:03:34.196 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-04T21:03:34.196 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.196 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.196 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.196 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s cmd 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' cmd' 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_list_parallel.log 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo cmd 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.197 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.198 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.198 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.198 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.198 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.198 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.198 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.198 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.198 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.198 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.198 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.198 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.198 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.198 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.198 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.199 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.200 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.201 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.202 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.202 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ list_parallel: /' 2024-04-04T21:03:34.202 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.202 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.202 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=cmd 2024-04-04T21:03:34.202 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41943 2024-04-04T21:03:34.202 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test cmd on pid 41943' 2024-04-04T21:03:34.202 INFO:tasks.workunit.client.0.smithi106.stdout:test cmd on pid 41943 2024-04-04T21:03:34.207 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=41943 2024-04-04T21:03:34.207 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-04T21:03:34.207 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.207 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s handler_error 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' handler_error' 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.208 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.209 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.210 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.210 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.210 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.210 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.210 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.210 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.210 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.210 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.210 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.210 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.210 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.210 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.210 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.210 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.211 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.211 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.211 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.211 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.211 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.211 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo handler_error 2024-04-04T21:03:34.211 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.211 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=handler_error 2024-04-04T21:03:34.211 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42000 2024-04-04T21:03:34.211 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test handler_error on pid 42000' 2024-04-04T21:03:34.212 INFO:tasks.workunit.client.0.smithi106.stdout:test handler_error on pid 42000 2024-04-04T21:03:34.220 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=42000 2024-04-04T21:03:34.220 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-04T21:03:34.220 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.220 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.220 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s io 2024-04-04T21:03:34.220 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' io' 2024-04-04T21:03:34.220 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo io 2024-04-04T21:03:34.220 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.220 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.220 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.220 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.220 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.221 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.221 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.221 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.221 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.221 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.221 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.221 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.221 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.221 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.221 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.221 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.221 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.221 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.221 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.222 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.223 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.224 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.225 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.226 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=io 2024-04-04T21:03:34.226 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.226 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.226 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42022 2024-04-04T21:03:34.226 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.226 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.226 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.226 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.226 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.226 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test io on pid 42022' 2024-04-04T21:03:34.226 INFO:tasks.workunit.client.0.smithi106.stdout:test io on pid 42022 2024-04-04T21:03:34.232 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=42022 2024-04-04T21:03:34.232 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.232 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.232 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.233 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.233 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.233 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-04T21:03:34.233 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.233 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.233 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s ec_io 2024-04-04T21:03:34.233 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.233 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' ec_io' 2024-04-04T21:03:34.233 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41830/exe 2024-04-04T21:03:34.233 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.233 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.233 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.234 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.234 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.234 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.234 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.234 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.234 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.234 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.234 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.234 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.234 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.234 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.234 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-04-04T21:03:34.234 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo ec_io 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.235 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.236 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.237 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.237 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.237 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.237 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.237 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.237 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.237 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.237 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=ec_io 2024-04-04T21:03:34.237 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42048 2024-04-04T21:03:34.237 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test ec_io on pid 42048' 2024-04-04T21:03:34.237 INFO:tasks.workunit.client.0.smithi106.stdout:test ec_io on pid 42048 2024-04-04T21:03:34.250 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=42048 2024-04-04T21:03:34.250 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-04T21:03:34.250 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.250 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.250 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.250 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.250 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.250 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.250 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.250 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.250 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.250 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s list 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' list' 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.251 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.252 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-04-04T21:03:34.253 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-04-04T21:03:34.253 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-04-04T21:03:34.253 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo list 2024-04-04T21:03:34.253 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-04-04T21:03:34.253 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.253 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.253 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.253 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.253 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.253 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.253 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.253 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.253 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.253 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.254 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.255 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.256 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.257 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.258 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.258 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.258 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.258 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.258 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.258 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.258 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.258 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.258 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.258 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.258 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.258 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.258 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.258 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41910/exe 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.259 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=list 2024-04-04T21:03:34.260 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42081 2024-04-04T21:03:34.261 INFO:tasks.workunit.client.0.smithi106.stdout:test list on pid 42081 2024-04-04T21:03:34.264 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test list on pid 42081' 2024-04-04T21:03:34.264 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=42081 2024-04-04T21:03:34.264 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-04T21:03:34.264 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.264 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.264 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41943/exe 2024-04-04T21:03:34.264 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/41863/exe 2024-04-04T21:03:34.264 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s ec_list 2024-04-04T21:03:34.264 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' ec_list' 2024-04-04T21:03:34.264 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.264 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.264 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.264 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.264 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-04-04T21:03:34.265 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.265 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.265 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.265 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.265 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.265 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.265 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.265 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.265 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.265 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.265 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.265 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.265 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.265 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.266 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.266 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.266 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.266 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.266 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:03:34.266 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:03:34.266 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.266 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36ee950640 4 set_mon_vals no callback set 2024-04-04T21:03:34.266 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36f2653780 5 monclient: authenticate success, global_id 4474 2024-04-04T21:03:34.266 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36f2653780 10 monclient: _renew_subs 2024-04-04T21:03:34.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36f2653780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:03:34.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559b3e720350 con 0x559b3e72a7e0 2024-04-04T21:03:34.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36f2653780 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:03:34.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36f2653780 10 monclient: _renew_subs 2024-04-04T21:03:34.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36f2653780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:03:34.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b3e72f640 con 0x559b3e72a7e0 2024-04-04T21:03:34.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36f2653780 1 librados: init done 2024-04-04T21:03:34.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:33.993+0000 7f36f2653780 10 librados: wait_for_osdmap waiting 2024-04-04T21:03:34.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.067+0000 7f36ed94e640 1 -- 172.21.15.106:0/1606434848 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f36e800fcd0 con 0x559b3e72a7e0 2024-04-04T21:03:34.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.068+0000 7f36ed94e640 1 --2- 172.21.15.106:0/1606434848 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f36dc06b0d0 0x7f36dc06d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:03:34.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.068+0000 7f36f0153640 1 --2- 172.21.15.106:0/1606434848 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f36dc06b0d0 0x7f36dc06d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:03:34.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.068+0000 7f36f0153640 10 monclient: get_auth_request con 0x7f36dc06b0d0 auth_method 0 2024-04-04T21:03:34.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.069+0000 7f36f0153640 1 --2- 172.21.15.106:0/1606434848 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f36dc06b0d0 0x7f36dc06d590 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x559b3e6590b0 tx=0x7f36e000c040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:03:34.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.069+0000 7f36ed94e640 10 client.4474.objecter ms_handle_connect 0x7f36dc06b0d0 2024-04-04T21:03:34.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.070+0000 7f36ed94e640 1 -- 172.21.15.106:0/1606434848 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f36e8055ac0 con 0x559b3e72a7e0 2024-04-04T21:03:34.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.070+0000 7f36ed94e640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_map(27..27 src has 1..27) v4 2024-04-04T21:03:34.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.070+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map got epochs [27,27] > 0 2024-04-04T21:03:34.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.070+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map decoding full epoch 27 2024-04-04T21:03:34.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.070+0000 7f36ed94e640 20 client.4474.objecter dump_active .. 0 homeless 2024-04-04T21:03:34.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.070+0000 7f36f2653780 10 librados: wait_for_osdmap done waiting 2024-04-04T21:03:34.275 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.275 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.275 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.275 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.275 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.275 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ cls: /' 2024-04-04T21:03:34.275 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.276 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.276 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.276 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.276 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.276 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.276 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-04-04T21:03:34.276 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_neorados_cmd.log 2024-04-04T21:03:34.276 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-04-04T21:03:34.276 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.276 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.276 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.276 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.276 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.276 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.277 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.278 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo ec_list 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_neorados_cls 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.279 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_neorados_cls.log 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=ec_list 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.280 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.281 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.281 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.281 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.281 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.281 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42105 2024-04-04T21:03:34.281 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-test ec_list on pid 42105 2024-04-04T21:03:34.289 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test ec_list on pid 42105' 2024-04-04T21:03:34.289 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=42105 2024-04-04T21:03:34.289 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-04T21:03:34.289 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.289 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.289 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.289 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.289 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.289 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.289 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.289 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s misc 2024-04-04T21:03:34.289 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' misc' 2024-04-04T21:03:34.289 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo misc 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_neorados_cmd 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.290 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ cmd: /' 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.291 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.292 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.293 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.294 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.295 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.295 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.295 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.295 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.295 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.295 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.295 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.295 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.295 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=misc 2024-04-04T21:03:34.295 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42126 2024-04-04T21:03:34.295 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test misc on pid 42126' 2024-04-04T21:03:34.295 INFO:tasks.workunit.client.0.smithi106.stdout:test misc on pid 42126 2024-04-04T21:03:34.307 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=42126 2024-04-04T21:03:34.307 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-04T21:03:34.307 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.307 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.307 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.307 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.307 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.307 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.307 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.307 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.307 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.307 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.308 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.308 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.308 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.308 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.308 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.308 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.308 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.308 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.308 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.308 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.308 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.308 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.308 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.308 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.309 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.310 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.310 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.310 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.310 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.310 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.310 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.310 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.310 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.310 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.310 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.310 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.310 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-04-04T21:03:34.310 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.310 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s pool 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' pool' 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.311 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.312 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.313 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.313 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.313 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.313 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.313 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.313 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.313 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.313 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.313 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.313 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.313 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.313 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.313 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.313 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo pool 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.314 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.315 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.316 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.316 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.316 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.316 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.316 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.316 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.316 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.316 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.316 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=pool 2024-04-04T21:03:34.316 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42159 2024-04-04T21:03:34.316 INFO:tasks.workunit.client.0.smithi106.stdout:test pool on pid 42159 2024-04-04T21:03:34.329 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test pool on pid 42159' 2024-04-04T21:03:34.330 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=42159 2024-04-04T21:03:34.330 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-04T21:03:34.330 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.330 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-04-04T21:03:34.330 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.330 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.330 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.330 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.330 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.330 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.330 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.330 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.330 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.330 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.331 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.332 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.333 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/42022/exe 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.334 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.335 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.336 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s read_operations 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.337 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.338 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.338 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.338 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.338 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' read_operations' 2024-04-04T21:03:34.338 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.338 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.338 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.338 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.338 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.338 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.338 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/42048/exe 2024-04-04T21:03:34.338 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.338 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo read_operations 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_neorados_io.log 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ io: /' 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.339 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.340 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.340 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_neorados_ec_io 2024-04-04T21:03:34.340 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ ec_io: /' 2024-04-04T21:03:34.340 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/42081/exe 2024-04-04T21:03:34.340 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=read_operations 2024-04-04T21:03:34.340 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.340 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.340 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.340 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.340 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.340 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.340 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42199 2024-04-04T21:03:34.340 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.341 INFO:tasks.workunit.client.0.smithi106.stdout:test read_operations on pid 42199 2024-04-04T21:03:34.358 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test read_operations on pid 42199' 2024-04-04T21:03:34.358 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.358 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=42199 2024-04-04T21:03:34.358 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.358 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-04T21:03:34.358 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.358 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.358 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.358 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.358 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.358 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.358 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.359 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.359 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.359 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.359 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_neorados_ec_io.log 2024-04-04T21:03:34.359 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.359 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.359 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.359 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.359 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.359 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.359 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.359 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.359 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.359 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/42000/exe 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_neorados_list.log 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ list: /' 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.360 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_neorados_list 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.361 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.362 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.362 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.362 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_neorados_io 2024-04-04T21:03:34.362 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.362 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.362 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.362 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.362 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.362 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.362 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.362 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.362 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.362 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.362 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s snapshots 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' snapshots' 2024-04-04T21:03:34.363 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.364 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.365 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.366 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.366 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.366 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.366 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.366 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.366 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.366 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_neorados_handler_error.log 2024-04-04T21:03:34.366 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo snapshots 2024-04-04T21:03:34.366 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.366 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.366 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.366 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.366 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.367 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.368 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.368 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.368 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.368 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.368 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.368 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_neorados_handler_error 2024-04-04T21:03:34.368 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.368 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.368 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.368 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.368 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.368 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.368 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.368 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.369 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.370 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.371 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.371 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.371 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.371 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.373 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.373 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/42105/exe 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ handler_error: /' 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.374 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.375 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.375 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.375 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.375 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.375 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.375 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.375 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=snapshots 2024-04-04T21:03:34.375 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42270 2024-04-04T21:03:34.375 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test snapshots on pid 42270' 2024-04-04T21:03:34.376 INFO:tasks.workunit.client.0.smithi106.stdout:test snapshots on pid 42270 2024-04-04T21:03:34.389 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=42270 2024-04-04T21:03:34.389 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-04T21:03:34.389 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.389 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_neorados_ec_list 2024-04-04T21:03:34.389 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.389 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.389 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.389 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.389 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.389 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.389 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.390 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.391 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/42126/exe 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.392 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.393 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.394 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.394 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.394 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.394 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.394 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.394 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.394 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.394 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.394 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.394 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s watch_notify 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' watch_notify' 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_neorados_misc 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ misc: /' 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_neorados_misc.log 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.395 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ ec_list: /' 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo watch_notify 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/42159/exe 2024-04-04T21:03:34.396 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_neorados_ec_list.log 2024-04-04T21:03:34.397 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.397 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.397 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.397 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.397 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.397 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.397 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.397 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.397 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.397 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.397 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.397 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.397 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.398 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_neorados_pool 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_neorados_pool.log 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=watch_notify 2024-04-04T21:03:34.399 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42322 2024-04-04T21:03:34.400 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test watch_notify on pid 42322' 2024-04-04T21:03:34.400 INFO:tasks.workunit.client.0.smithi106.stdout:test watch_notify on pid 42322 2024-04-04T21:03:34.413 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=42322 2024-04-04T21:03:34.413 INFO:tasks.workunit.client.0.smithi106.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-04T21:03:34.413 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.413 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-04-04T21:03:34.413 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.413 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.413 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.413 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.413 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.413 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.413 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.413 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ pool: /' 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:++ printf %25s write_operations 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.414 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:+ r=' write_operations' 2024-04-04T21:03:34.415 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:++ echo write_operations 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.416 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:++ awk '{print $1}' 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.417 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.418 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.418 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.418 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.418 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.418 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.418 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.418 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.418 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.418 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.418 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.418 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.418 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.418 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.418 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.419 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.420 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.421 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.422 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.423 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/42199/exe 2024-04-04T21:03:34.424 INFO:tasks.workunit.client.0.smithi106.stderr:+ ff=write_operations 2024-04-04T21:03:34.424 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42359 2024-04-04T21:03:34.424 INFO:tasks.workunit.client.0.smithi106.stderr:+ echo 'test write_operations on pid 42359' 2024-04-04T21:03:34.424 INFO:tasks.workunit.client.0.smithi106.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-04-04T21:03:34.424 INFO:tasks.workunit.client.0.smithi106.stdout:test write_operations on pid 42359 2024-04-04T21:03:34.441 INFO:tasks.workunit.client.0.smithi106.stderr:+ pids[$f]=42359 2024-04-04T21:03:34.441 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.441 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.442 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.442 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.442 INFO:tasks.workunit.client.0.smithi106.stderr:+ ret=0 2024-04-04T21:03:34.442 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' 1 -eq 1 ']' 2024-04-04T21:03:34.442 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/42270/exe 2024-04-04T21:03:34.442 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.442 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:03:34.442 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41284 2024-04-04T21:03:34.442 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41284 2024-04-04T21:03:34.442 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.442 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.442 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.442 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.442 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -f /etc/bashrc ']' 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:+ . /etc/bashrc 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -z '' ']' 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:++ BASHRCSOURCED=Y 2024-04-04T21:03:34.443 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:++ shopt -q login_shell 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:+++ umask 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.444 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' 0022 -eq 0 ']' 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:++ SHELL=/bin/bash 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.445 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-04T21:03:34.446 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.446 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-04T21:03:34.446 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.446 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-04T21:03:34.446 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.446 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_neorados_read_operations 2024-04-04T21:03:34.446 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-04T21:03:34.446 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.446 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-04T21:03:34.446 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.446 INFO:tasks.workunit.client.0.smithi106.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-04T21:03:34.446 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.446 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-04T21:03:34.446 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ read_operations: /' 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_neorados_read_operations.log 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_neorados_snapshots.log 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ snapshots: /' 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'grep=grep --color=auto' 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorls.sh 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' '!' -t 0 ']' 2024-04-04T21:03:34.447 INFO:tasks.workunit.client.0.smithi106.stderr:+++ return 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export S_COLORS=auto 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:+++ S_COLORS=auto 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_neorados_snapshots 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.448 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.449 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/42322/exe 2024-04-04T21:03:34.450 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/gawk.sh 2024-04-04T21:03:34.451 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.452 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.452 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.452 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.452 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.452 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.452 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-04T21:03:34.452 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.452 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.452 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/lang.sh 2024-04-04T21:03:34.452 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.452 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup 2024-04-04T21:03:34.452 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.452 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_neorados_watch_notify 2024-04-04T21:03:34.452 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/locale 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ watch_notify: /' 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:++++ export LANG=en_US.UTF-8 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:++++ LANG=en_US.UTF-8 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.453 INFO:tasks.workunit.client.0.smithi106.stderr:+++ LANG=en_US.UTF-8 2024-04-04T21:03:34.454 INFO:tasks.workunit.client.0.smithi106.stderr:+++ unset LANG_backup config 2024-04-04T21:03:34.454 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n '' ']' 2024-04-04T21:03:34.454 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-04T21:03:34.454 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' dumb = linux ']' 2024-04-04T21:03:34.454 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.454 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-04T21:03:34.454 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.454 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/less.sh 2024-04-04T21:03:34.454 INFO:tasks.workunit.client.0.smithi106.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-04T21:03:34.454 INFO:tasks.workunit.client.0.smithi106.stderr:++ for i in /etc/profile.d/*.sh 2024-04-04T21:03:34.454 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-04T21:03:34.454 INFO:tasks.workunit.client.0.smithi106.stderr:++ '[' '' ']' 2024-04-04T21:03:34.454 INFO:tasks.workunit.client.0.smithi106.stderr:++ . /etc/profile.d/which2.sh 2024-04-04T21:03:34.454 INFO:tasks.workunit.client.0.smithi106.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-04T21:03:34.455 INFO:tasks.workunit.client.0.smithi106.stderr:+++++ readlink /proc/42359/exe 2024-04-04T21:03:34.455 INFO:tasks.workunit.client.0.smithi106.stderr:++++ basename /usr/bin/bash 2024-04-04T21:03:34.455 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_declare='declare -f' 2024-04-04T21:03:34.455 INFO:tasks.workunit.client.0.smithi106.stderr:+++ which_opt=-f 2024-04-04T21:03:34.455 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export which_declare 2024-04-04T21:03:34.455 INFO:tasks.workunit.client.0.smithi106.stderr:+++ export -f which 2024-04-04T21:03:34.455 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset i 2024-04-04T21:03:34.455 INFO:tasks.workunit.client.0.smithi106.stderr:++ unset -f pathmunge 2024-04-04T21:03:34.455 INFO:tasks.workunit.client.0.smithi106.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-04T21:03:34.455 INFO:tasks.workunit.client.0.smithi106.stderr:+ export PATH 2024-04-04T21:03:34.455 INFO:tasks.workunit.client.0.smithi106.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-04T21:03:34.455 INFO:tasks.workunit.client.0.smithi106.stderr:+ unset rc 2024-04-04T21:03:34.455 INFO:tasks.workunit.client.0.smithi106.stderr:+ ceph_test_neorados_write_operations 2024-04-04T21:03:34.455 INFO:tasks.workunit.client.0.smithi106.stderr:+ sed 's/^/ write_operations: /' 2024-04-04T21:03:34.456 INFO:tasks.workunit.client.0.smithi106.stderr:+ tee ceph_test_neorados_write_operations.log 2024-04-04T21:03:35.636 INFO:tasks.workunit.client.0.smithi106.stdout:04T21:03:34.070+0000 7f36f2653780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:03:35.636 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.070+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f36e40086d0 con 0x559b3e72a7e0 2024-04-04T21:03:35.636 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.142+0000 7f36ed94e640 1 -- 172.21.15.106:0/1606434848 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36e8053cc0 con 0x559b3e72a7e0 2024-04-04T21:03:35.636 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.142+0000 7f36ed94e640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-04-04T21:03:35.636 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.142+0000 7f36ee14f640 10 client.4474.objecter _get_latest_version latest 27, have it 2024-04-04T21:03:35.637 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.142+0000 7f36f2653780 10 client.4474.objecter create_pool name=ceph_test_rados_api_asio 2024-04-04T21:03:35.637 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.142+0000 7f36f2653780 10 client.4474.objecter pool_op_submit 1 2024-04-04T21:03:35.637 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.142+0000 7f36f2653780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:03:35.637 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:34.142+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x559b3e60c3d0 con 0x559b3e72a7e0 2024-04-04T21:03:35.637 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.602+0000 7f36ed94e640 1 -- 172.21.15.106:0/1606434848 <== mon.1 v2:172.21.15.203:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v28) v1 ==== 43+0+0 (secure 0 0 0) 0x7f36e8057a20 con 0x559b3e72a7e0 2024-04-04T21:03:35.637 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.602+0000 7f36ed94e640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 pool_op_reply(tid 1 (0) Success v28) v1 2024-04-04T21:03:35.637 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.602+0000 7f36ed94e640 10 client.4474.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v28) v1 2024-04-04T21:03:35.637 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.602+0000 7f36ed94e640 10 client.4474.objecter have request 1 at 0x559b3e72d470 Op: create 2024-04-04T21:03:35.637 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.602+0000 7f36ed94e640 20 client.4474.objecter waiting for client to reach epoch 28 before calling back 2024-04-04T21:03:35.637 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.602+0000 7f36ed94e640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:03:35.637 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.602+0000 7f36ed94e640 10 monclient: _renew_subs 2024-04-04T21:03:35.637 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.602+0000 7f36ed94e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:03:35.637 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.602+0000 7f36ed94e640 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=28}) v3 -- 0x7f36dc070e70 con 0x559b3e72a7e0 2024-04-04T21:03:35.637 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.602+0000 7f36ed94e640 10 client.4474.objecter done 2024-04-04T21:03:35.638 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.635+0000 7f36ed94e640 1 -- 172.21.15.106:0/1606434848 <== mon.1 v2:172.21.15.203:3300/0 8 ==== osd_map(28..28 src has 1..28) v4 ==== 5461+0+0 (secure 0 0 0) 0x7f36e8085030 con 0x559b3e72a7e0 2024-04-04T21:03:35.638 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.635+0000 7f36ed94e640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_map(28..28 src has 1..28) v4 2024-04-04T21:03:35.638 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.635+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map got epochs [28,28] > 27 2024-04-04T21:03:35.638 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.635+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map decoding incremental epoch 28 2024-04-04T21:03:35.638 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36ed94e640 20 client.4474.objecter dump_active .. 0 homeless 2024-04-04T21:03:35.638 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 10 librados: set snap read head -> 1 2024-04-04T21:03:35.638 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 10 librados: writefull oid=exist nspace= 2024-04-04T21:03:38.868 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:02:30.385 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:38.877 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:02:30.394 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:38.880 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e72d670 2024-04-04T21:03:38.880 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 28 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:38.880 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:38.880 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:38.880 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 1 --2- 172.21.15.106:0/1606434848 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x559b3e7cd950 0x559b3e7cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:03:38.880 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:38.881 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 10 client.4474.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 2 osd.7 2024-04-04T21:03:38.881 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:38.881 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 2 2024-04-04T21:03:38.881 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 15 client.4474.objecter _send_op 2 to 9.1 on osd.7 2024-04-04T21:03:38.881 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:2 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) v9 -- 0x559b3e7d0280 con 0x559b3e7cd950 2024-04-04T21:03:38.881 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:38.881 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.636+0000 7f36f2653780 5 client.4474.objecter 1 in flight 2024-04-04T21:03:38.881 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.663+0000 7f36ef952640 1 --2- 172.21.15.106:0/1606434848 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x559b3e7cd950 0x559b3e7cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:03:38.881 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.663+0000 7f36ef952640 10 monclient: get_auth_request con 0x559b3e7cd950 auth_method 0 2024-04-04T21:03:38.881 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.666+0000 7f36ed94e640 10 client.4474.objecter ms_handle_connect 0x559b3e7cd950 2024-04-04T21:03:38.881 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:35.666+0000 7f36ef952640 1 --2- 172.21.15.106:0/1606434848 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x559b3e7cd950 0x559b3e7cfd40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:03:38.881 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:36.879+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:38.881 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:36.879+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:38.881 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:36.879+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:38.882 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:36.879+0000 7f36ec94c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:03:38.882 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:37.879+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:38.882 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:37.879+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:38.882 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:37.879+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:38.882 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:38.879+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:38.882 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:38.879+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:38.928 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:02:30.444 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:38.933 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:02:30.450 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:38.984 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:02:30.500 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:38.999 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:02:30.516 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:39.039 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:30.521 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:39.048 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:30.530 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:39.105 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:30.588 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:03:45.121 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:38.879+0000 7f36ec94c640 20 monclient: _check_auth_rotating api_service: [==========] Running 4 tests from 1 test suite. 2024-04-04T21:03:45.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: [----------] Global test environment set-up. 2024-04-04T21:03:45.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: [----------] 4 tests from LibRadosService 2024-04-04T21:03:45.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-04-04T21:03:45.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5489 ms) 2024-04-04T21:03:45.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-04-04T21:03:45.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: [ OK ] LibRadosService.RegisterLate (54 ms) 2024-04-04T21:03:45.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-04-04T21:03:45.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: cluster: 2024-04-04T21:03:45.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: id: 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:03:45.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: health: HEALTH_WARN 2024-04-04T21:03:45.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 14 pool(s) do not have an application enabled 2024-04-04T21:03:45.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 2024-04-04T21:03:45.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: services: 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: osd: 8 osds: 8 up (since 83s), 8 in (since 2m) 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: data: 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: pools: 16 pools, 457 pgs 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: objects: 2 objects, 449 KiB 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: pgs: 98.031% pgs unknown 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 448 unknown 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 9 active+clean 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 2024-04-04T21:03:45.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: cluster: 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: id: 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: health: HEALTH_WARN 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 17 pool(s) do not have an application enabled 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: services: 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: osd: 8 osds: 8 up (since 85s), 8 in (since 2m) 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: foo: 12 portals active (1 hosts, 3 zones) 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: data: 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: pools: 37 pools, 1057 pgs 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: objects: 2 objects, 449 KiB 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-04-04T21:03:45.124 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: pgs: 53.075% pgs unknown 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 44.087% pgs not active 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 561 unknown 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 353 creating+peering 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 113 creating+activating 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 30 active+clean 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: cluster: 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: id: 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: health: HEALTH_WARN 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 18 pool(s) do not have an application enabled 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: services: 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-04-04T21:03:45.125 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-04-04T21:03:45.126 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: osd: 8 osds: 8 up (since 88s), 8 in (since 2m) 2024-04-04T21:03:45.126 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-04-04T21:03:45.126 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-04-04T21:03:45.126 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 2024-04-04T21:03:45.126 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: data: 2024-04-04T21:03:45.126 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: pools: 38 pools, 1089 pgs 2024-04-04T21:03:45.126 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: objects: 2 objects, 449 KiB 2024-04-04T21:03:45.126 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-04-04T21:03:45.126 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: pgs: 44.169% pgs unknown 2024-04-04T21:03:45.126 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 53.076% pgs not active 2024-04-04T21:03:45.126 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 481 unknown 2024-04-04T21:03:48.996 INFO:tasks.workunit.client.0.smithi106.stdout: not needed by client.admin 2024-04-04T21:03:48.997 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:38.994+0000 7f36f23b7640 10 client.4474.objecter tick 2024-04-04T21:03:48.997 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:39.881+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:48.997 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:39.881+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:48.997 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:39.881+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:48.997 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:40.882+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:48.997 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:40.882+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:48.997 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:40.882+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:48.997 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:41.882+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:48.997 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:41.882+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:48.997 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:41.882+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:48.997 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:42.882+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:48.997 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:42.882+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:48.997 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:42.882+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:48.997 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:43.882+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:48.998 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:43.882+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:48.998 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:43.882+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:48.998 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:43.995+0000 7f36f23b7640 10 client.4474.objecter tick 2024-04-04T21:03:48.998 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:44.882+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:48.998 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:44.882+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:48.998 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:44.882+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:48.998 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:45.882+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:48.998 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:45.882+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:48.998 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:45.882+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:48.998 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:46.882+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:48.998 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:46.882+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:48.998 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:46.882+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:48.998 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:46.882+0000 7f36ec94c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:03:48.999 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:47.882+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:48.999 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:47.882+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:48.999 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:47.882+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:48.999 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:48.882+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:48.999 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:48.882+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:48.999 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:48.882+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:48.999 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:48.996+0000 7f36f23b7640 10 client.4474.objecter tick 2024-04-04T21:03:48.999 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:48.996+0000 7f36f23b7640 2 client.4474.objecter tid 2 on osd.7 is laggy 2024-04-04T21:03:48.999 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:48.996+0000 7f36f23b7640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:03:48.999 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:48.996+0000 7f36f23b7640 10 monclient: _renew_subs 2024-04-04T21:03:49.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:48.996+0000 7f36f23b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:03:49.269 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:48.996+0000 7f36f23b7640 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=29}) v3 -- 0x7f36c8002f90 con 0x559b3e72a7e0 2024-04-04T21:03:49.269 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:48.996+0000 7f36f23b7640 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- ping magic: 0 v1 -- 0x7f36c8003360 con 0x559b3e7cd950 2024-04-04T21:03:49.269 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 1 -- 172.21.15.106:0/1606434848 <== mon.1 v2:172.21.15.203:3300/0 9 ==== osd_map(29..37 src has 1..37) v4 ==== 29172+0+0 (secure 0 0 0) 0x7f36e8074030 con 0x559b3e72a7e0 2024-04-04T21:03:49.269 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_map(29..37 src has 1..37) v4 2024-04-04T21:03:49.269 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map got epochs [29,37] > 28 2024-04-04T21:03:49.269 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map decoding incremental epoch 29 2024-04-04T21:03:49.269 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 10 client.4474.objecter checking op 2 2024-04-04T21:03:49.269 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 20 client.4474.objecter _calc_target epoch 29 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:49.269 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:49.269 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map decoding incremental epoch 30 2024-04-04T21:03:49.269 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 10 client.4474.objecter checking op 2 2024-04-04T21:03:49.269 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 20 client.4474.objecter _calc_target epoch 30 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:49.269 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:49.269 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map decoding incremental epoch 31 2024-04-04T21:03:49.270 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 10 client.4474.objecter checking op 2 2024-04-04T21:03:49.270 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 20 client.4474.objecter _calc_target epoch 31 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:49.270 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:49.270 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map decoding incremental epoch 32 2024-04-04T21:03:49.270 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 10 client.4474.objecter checking op 2 2024-04-04T21:03:49.270 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 20 client.4474.objecter _calc_target epoch 32 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:49.270 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.267+0000 7f36ed94e640 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:49.270 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map decoding incremental epoch 33 2024-04-04T21:03:49.270 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 10 client.4474.objecter checking op 2 2024-04-04T21:03:49.270 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 20 client.4474.objecter _calc_target epoch 33 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:49.270 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:49.270 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map decoding incremental epoch 34 2024-04-04T21:03:49.270 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 10 client.4474.objecter checking op 2 2024-04-04T21:03:52.883 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 20 client.4474.objecter _calc_target epoch 34 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:52.884 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:52.884 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map decoding incremental epoch 35 2024-04-04T21:03:52.884 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 10 client.4474.objecter checking op 2 2024-04-04T21:03:52.884 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 20 client.4474.objecter _calc_target epoch 35 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:52.884 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:52.884 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map decoding incremental epoch 36 2024-04-04T21:03:52.884 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 10 client.4474.objecter checking op 2 2024-04-04T21:03:52.884 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 20 client.4474.objecter _calc_target epoch 36 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:52.884 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:52.884 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map decoding incremental epoch 37 2024-04-04T21:03:52.884 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 10 client.4474.objecter checking op 2 2024-04-04T21:03:52.884 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 20 client.4474.objecter _calc_target epoch 37 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:52.884 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:52.885 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 20 client.4474.objecter dump_active .. 0 homeless 2024-04-04T21:03:52.885 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.268+0000 7f36ed94e640 20 client.4474.objecter 2 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-04-04T21:03:52.885 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.882+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:52.885 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.882+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:52.885 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:49.882+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:52.885 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:50.882+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:52.885 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:50.882+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:52.885 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:50.882+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:52.885 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:51.236+0000 7f36ed94e640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 1 ==== osd_map(29..37 src has 1..37) v4 ==== 29172+0+0 (crc 0 0 0) 0x7f36e4011040 con 0x559b3e7cd950 2024-04-04T21:03:52.885 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:51.236+0000 7f36ed94e640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_map(29..37 src has 1..37) v4 2024-04-04T21:03:52.885 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:51.236+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map ignoring epochs [29,37] <= 37 2024-04-04T21:03:52.885 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:51.236+0000 7f36ed94e640 20 client.4474.objecter dump_active .. 0 homeless 2024-04-04T21:03:52.886 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:51.236+0000 7f36ed94e640 20 client.4474.objecter 2 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-04-04T21:03:52.886 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:51.882+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:52.886 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:51.882+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:52.886 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:51.882+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:52.886 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:52.883+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:56.420 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:52.883+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:56.421 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:52.883+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:56.421 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:53.883+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:56.421 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:53.883+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:56.421 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:53.883+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:56.421 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:53.997+0000 7f36f23b7640 10 client.4474.objecter tick 2024-04-04T21:03:56.421 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:53.997+0000 7f36f23b7640 2 client.4474.objecter tid 2 on osd.7 is laggy 2024-04-04T21:03:56.421 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:53.997+0000 7f36f23b7640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:03:56.421 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:53.997+0000 7f36f23b7640 10 monclient: _renew_subs 2024-04-04T21:03:56.421 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:53.997+0000 7f36f23b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:03:56.421 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:53.997+0000 7f36f23b7640 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=38}) v3 -- 0x7f36c8003560 con 0x559b3e72a7e0 2024-04-04T21:03:56.421 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:53.997+0000 7f36f23b7640 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- ping magic: 0 v1 -- 0x7f36c8003790 con 0x559b3e7cd950 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:54.000+0000 7f36ed94e640 1 -- 172.21.15.106:0/1606434848 <== mon.1 v2:172.21.15.203:3300/0 10 ==== osd_map(38..38 src has 1..38) v4 ==== 2128+0+0 (secure 0 0 0) 0x7f36e8087320 con 0x559b3e72a7e0 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:54.000+0000 7f36ed94e640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_map(38..38 src has 1..38) v4 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:54.000+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map got epochs [38,38] > 37 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:54.000+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map decoding incremental epoch 38 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:54.000+0000 7f36ed94e640 10 client.4474.objecter checking op 2 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:54.000+0000 7f36ed94e640 20 client.4474.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:54.000+0000 7f36ed94e640 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:54.000+0000 7f36ed94e640 20 client.4474.objecter dump_active .. 0 homeless 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:54.000+0000 7f36ed94e640 20 client.4474.objecter 2 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:54.883+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:54.883+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:54.883+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:55.883+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:55.883+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:56.422 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:55.883+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 2 ==== osd_op_reply(2 exist [writefull 0~5] v37'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(2 exist [writefull 0~5] v37'1 uv1 ondisk = 0) v8 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 2 ondisk uv 1 in 9.1 attempt 0 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 0 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 2 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36ef952640 15 client.4474.objecter _finish_op 2 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 2 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36ef952640 5 client.4474.objecter 0 in flight 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 10 librados: Objecter returned from writefull r=0 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e72d670 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-04-04T21:03:56.423 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:56.424 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:56.424 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:56.424 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 10 client.4474.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 3 osd.7 2024-04-04T21:03:56.424 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:56.424 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 3 2024-04-04T21:03:56.424 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 15 client.4474.objecter _send_op 3 to 9.1 on osd.7 2024-04-04T21:03:56.424 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:3 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e7c6600 con 0x559b3e7cd950 2024-04-04T21:03:56.424 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:56.424 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 5 client.4474.objecter 1 in flight 2024-04-04T21:03:56.424 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e7c69c0 2024-04-04T21:03:56.424 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-04-04T21:03:56.424 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 20 client.4474.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-04-04T21:03:56.424 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:56.424 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:56.425 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 10 client.4474.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 4 osd.7 2024-04-04T21:03:56.425 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:56.425 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 4 2024-04-04T21:03:56.425 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 15 client.4474.objecter _send_op 4 to 9.1 on osd.7 2024-04-04T21:03:56.489 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:4 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e7c6e70 con 0x559b3e7cd950 2024-04-04T21:03:56.490 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 5 2024-04-04T21:03:56.490 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.420+0000 7f36f2653780 5 client.4474.objecter 2 in flight 2024-04-04T21:03:56.490 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.480+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:56.490 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.480+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-04-04T21:03:56.490 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.480+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:56.490 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.480+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 3 ondisk uv 1 in 9.1 attempt 0 2024-04-04T21:03:56.490 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.480+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 5 2024-04-04T21:03:56.490 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.480+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 3 2024-04-04T21:03:56.490 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.480+0000 7f36ef952640 15 client.4474.objecter _finish_op 3 2024-04-04T21:03:56.490 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.480+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 5 2024-04-04T21:03:56.490 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.480+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 3 2024-04-04T21:03:56.490 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.480+0000 7f36ef952640 5 client.4474.objecter 1 in flight 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 4 ondisk uv 0 in 9.1 attempt 0 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36ef952640 10 client.4474.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 0 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 4 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36ef952640 15 client.4474.objecter _finish_op 4 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 4 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36ef952640 5 client.4474.objecter 0 in flight 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (67 ms) 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e7c69c0 2024-04-04T21:03:56.491 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-04-04T21:03:56.502 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:56.502 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:56.502 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:56.502 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 10 client.4474.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 5 osd.7 2024-04-04T21:03:56.502 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:56.503 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 5 2024-04-04T21:03:56.503 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 15 client.4474.objecter _send_op 5 to 9.1 on osd.7 2024-04-04T21:03:56.503 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:5 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e7c7240 con 0x559b3e7cd950 2024-04-04T21:03:56.503 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:56.503 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 5 client.4474.objecter 1 in flight 2024-04-04T21:03:56.503 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e72d670 2024-04-04T21:03:56.503 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-04-04T21:03:56.503 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 20 client.4474.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-04-04T21:03:56.503 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:56.503 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:56.503 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 10 client.4474.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 6 osd.7 2024-04-04T21:03:56.503 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:56.503 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 6 2024-04-04T21:03:56.503 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 15 client.4474.objecter _send_op 6 to 9.1 on osd.7 2024-04-04T21:03:56.504 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:6 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e7d0620 con 0x559b3e7cd950 2024-04-04T21:03:56.504 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 5 2024-04-04T21:03:56.504 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.487+0000 7f36f2653780 5 client.4474.objecter 2 in flight 2024-04-04T21:03:56.504 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:56.504 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-04-04T21:03:56.504 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:56.504 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 5 ondisk uv 1 in 9.1 attempt 0 2024-04-04T21:03:56.504 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 5 2024-04-04T21:03:56.504 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 5 2024-04-04T21:03:56.504 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 15 client.4474.objecter _finish_op 5 2024-04-04T21:03:56.505 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 5 2024-04-04T21:03:56.505 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 5 2024-04-04T21:03:56.505 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 5 client.4474.objecter 1 in flight 2024-04-04T21:03:56.505 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:56.505 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-04-04T21:03:56.505 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:56.505 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 6 ondisk uv 0 in 9.1 attempt 0 2024-04-04T21:03:56.505 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 10 client.4474.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-04T21:03:56.505 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 0 2024-04-04T21:03:56.505 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 6 2024-04-04T21:03:56.505 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 15 client.4474.objecter _finish_op 6 2024-04-04T21:03:56.505 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:56.505 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 6 2024-04-04T21:03:56.505 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36ef952640 5 client.4474.objecter 0 in flight 2024-04-04T21:03:56.506 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (14 ms) 2024-04-04T21:03:56.506 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-04-04T21:03:56.506 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e72d670 2024-04-04T21:03:56.506 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-04-04T21:03:56.506 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36f2653780 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:56.506 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:56.506 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:56.506 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36f2653780 10 client.4474.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 7 osd.7 2024-04-04T21:03:56.506 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:56.506 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 7 2024-04-04T21:03:56.506 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36f2653780 15 client.4474.objecter _send_op 7 to 9.1 on osd.7 2024-04-04T21:03:56.506 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.501+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:7 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e7c69a0 con 0x559b3e7cd950 2024-04-04T21:03:56.513 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.502+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:56.514 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.502+0000 7f36f2653780 5 client.4474.objecter 1 in flight 2024-04-04T21:03:56.514 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.502+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e811050 2024-04-04T21:03:56.514 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.502+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-04-04T21:03:56.514 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.502+0000 7f36f2653780 20 client.4474.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-04-04T21:03:56.514 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.502+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:56.514 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.502+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:56.514 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.502+0000 7f36f2653780 10 client.4474.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 8 osd.7 2024-04-04T21:03:56.514 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.502+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:56.514 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.502+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 8 2024-04-04T21:03:56.514 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.502+0000 7f36f2653780 15 client.4474.objecter _send_op 8 to 9.1 on osd.7 2024-04-04T21:03:56.514 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.502+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:8 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e811500 con 0x559b3e7cd950 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.502+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 5 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.502+0000 7f36f2653780 5 client.4474.objecter 2 in flight 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 7 ondisk uv 1 in 9.1 attempt 0 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 5 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 7 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 15 client.4474.objecter _finish_op 7 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 5 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 7 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 5 client.4474.objecter 1 in flight 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-04-04T21:03:56.515 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:57.016 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 8 ondisk uv 0 in 9.1 attempt 0 2024-04-04T21:03:57.016 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 10 client.4474.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-04T21:03:57.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 0 2024-04-04T21:03:57.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (12 ms) 2024-04-04T21:03:57.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 8 2024-04-04T21:03:57.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 15 client.4474.objecter _finish_op 8 2024-04-04T21:03:57.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-04-04T21:03:57.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 8 2024-04-04T21:03:57.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36ef952640 5 client.4474.objecter 0 in flight 2024-04-04T21:03:57.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-04-04T21:03:57.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 20 librados: queue_aio_write 0x559b3e5c8830 completion 0x559b3e5a47b0 write_seq 1 2024-04-04T21:03:57.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e72d670 2024-04-04T21:03:57.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:57.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:57.017 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:57.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:57.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 10 client.4474.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 9 osd.7 2024-04-04T21:03:57.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:57.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 9 2024-04-04T21:03:57.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 15 client.4474.objecter _send_op 9 to 9.1 on osd.7 2024-04-04T21:03:57.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:9 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e7d19d0 con 0x559b3e7cd950 2024-04-04T21:03:57.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 5 client.4474.objecter 1 in flight 2024-04-04T21:03:57.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.513+0000 7f36f2653780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-04-04T21:03:57.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.884+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:57.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.884+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:57.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.884+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:57.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:56.884+0000 7f36ec94c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:03:57.018 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (502 ms) 2024-04-04T21:03:57.019 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-04-04T21:03:57.068 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 9 ==== osd_op_reply(9 exist [write 0~5] v38'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:57.068 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(9 exist [write 0~5] v38'2 uv2 ondisk = 0) v8 2024-04-04T21:03:57.068 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:57.068 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 9 ondisk uv 2 in 9.1 attempt 0 2024-04-04T21:03:57.068 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 0 2024-04-04T21:03:57.068 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 9 2024-04-04T21:03:57.068 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36ef952640 15 client.4474.objecter _finish_op 9 2024-04-04T21:03:57.069 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.069 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 9 2024-04-04T21:03:57.069 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36ef952640 5 client.4474.objecter 0 in flight 2024-04-04T21:03:57.069 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36ef952640 20 librados: complete_aio_write 0x559b3e5a47b0 2024-04-04T21:03:57.069 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-04-04T21:03:57.069 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 20 librados: queue_aio_write 0x559b3e5c8830 completion 0x559b3e5ad480 write_seq 2 2024-04-04T21:03:57.069 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e72d670 2024-04-04T21:03:57.069 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:57.069 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:57.069 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:57.069 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:57.069 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 10 client.4474.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 10 osd.7 2024-04-04T21:03:57.069 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:57.069 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 10 2024-04-04T21:03:57.070 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 15 client.4474.objecter _send_op 10 to 9.1 on osd.7 2024-04-04T21:03:57.070 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:10 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e7d19d0 con 0x559b3e7cd950 2024-04-04T21:03:57.070 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.070 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 5 client.4474.objecter 1 in flight 2024-04-04T21:03:57.070 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.016+0000 7f36f2653780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-04-04T21:03:57.070 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.068+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 10 ==== osd_op_reply(10 exist [write 0~5] v38'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:57.070 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.068+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(10 exist [write 0~5] v38'3 uv3 ondisk = 0) v8 2024-04-04T21:03:57.070 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.068+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:57.104 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.068+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 10 ondisk uv 3 in 9.1 attempt 0 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.068+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 0 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.068+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 10 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.068+0000 7f36ef952640 15 client.4474.objecter _finish_op 10 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.068+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.068+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 10 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.068+0000 7f36ef952640 5 client.4474.objecter 0 in flight 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.068+0000 7f36ef952640 20 librados: complete_aio_write 0x559b3e5ad480 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (53 ms) 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 20 librados: queue_aio_write 0x559b3e5c8830 completion 0x559b3e7d1b30 write_seq 3 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e72d670 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 10 client.4474.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 11 osd.7 2024-04-04T21:03:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 11 2024-04-04T21:03:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 15 client.4474.objecter _send_op 11 to 9.1 on osd.7 2024-04-04T21:03:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:11 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e7f4170 con 0x559b3e7cd950 2024-04-04T21:03:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 5 client.4474.objecter 1 in flight 2024-04-04T21:03:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.069+0000 7f36f2653780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-04-04T21:03:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 11 ==== osd_op_reply(11 exist [write 0~5] v38'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(11 exist [write 0~5] v38'4 uv4 ondisk = 0) v8 2024-04-04T21:03:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 11 ondisk uv 4 in 9.1 attempt 0 2024-04-04T21:03:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 0 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 11 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36ef952640 15 client.4474.objecter _finish_op 11 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 11 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36ef952640 5 client.4474.objecter 0 in flight 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36ef952640 20 librados: complete_aio_write 0x559b3e7d1b30 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (36 ms) 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e72d670 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 10 client.4474.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 12 osd.7 2024-04-04T21:03:57.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:57.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 12 2024-04-04T21:03:57.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 15 client.4474.objecter _send_op 12 to 9.1 on osd.7 2024-04-04T21:03:57.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:12 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e7d09c0 con 0x559b3e7cd950 2024-04-04T21:03:57.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 5 client.4474.objecter 1 in flight 2024-04-04T21:03:57.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e7d0d60 2024-04-04T21:03:57.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-04-04T21:03:57.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 20 client.4474.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-04-04T21:03:57.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:57.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 10 client.4474.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 13 osd.7 2024-04-04T21:03:57.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 13 2024-04-04T21:03:57.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 15 client.4474.objecter _send_op 13 to 9.1 on osd.7 2024-04-04T21:03:57.121 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:13 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e7d1210 con 0x559b3e7cd950 2024-04-04T21:03:57.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 5 2024-04-04T21:03:57.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.105+0000 7f36f2653780 5 client.4474.objecter 2 in flight 2024-04-04T21:03:57.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (17 ms) 2024-04-04T21:03:57.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-04-04T21:03:57.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:57.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-04-04T21:03:57.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:57.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 12 ondisk uv 4 in 9.1 attempt 0 2024-04-04T21:03:57.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 5 2024-04-04T21:03:57.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 12 2024-04-04T21:03:57.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 15 client.4474.objecter _finish_op 12 2024-04-04T21:03:57.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 5 2024-04-04T21:03:57.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 12 2024-04-04T21:03:57.122 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 5 client.4474.objecter 1 in flight 2024-04-04T21:03:57.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:57.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-04-04T21:03:57.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:57.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 13 ondisk uv 0 in 9.1 attempt 0 2024-04-04T21:03:57.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 10 client.4474.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-04T21:03:57.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 0 2024-04-04T21:03:57.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 13 2024-04-04T21:03:57.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 15 client.4474.objecter _finish_op 13 2024-04-04T21:03:57.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 13 2024-04-04T21:03:57.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36ef952640 5 client.4474.objecter 0 in flight 2024-04-04T21:03:57.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e7d09c0 2024-04-04T21:03:57.123 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-04-04T21:03:57.135 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:57.135 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:57.135 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:57.135 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 10 client.4474.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 14 osd.7 2024-04-04T21:03:57.135 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:57.135 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 14 2024-04-04T21:03:57.135 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 15 client.4474.objecter _send_op 14 to 9.1 on osd.7 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:14 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e7d0e70 con 0x559b3e7cd950 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 5 client.4474.objecter 1 in flight 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e7d1210 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 20 client.4474.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 10 client.4474.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 15 osd.7 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 15 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 15 client.4474.objecter _send_op 15 to 9.1 on osd.7 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:15 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e721fd0 con 0x559b3e7cd950 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 5 2024-04-04T21:03:57.136 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.122+0000 7f36f2653780 5 client.4474.objecter 2 in flight 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (13 ms) 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 14 ondisk uv 4 in 9.1 attempt 0 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 5 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 14 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 15 client.4474.objecter _finish_op 14 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 5 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 14 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 5 client.4474.objecter 1 in flight 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-04-04T21:03:57.137 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 15 ondisk uv 0 in 9.1 attempt 0 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 10 client.4474.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 0 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 15 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 15 client.4474.objecter _finish_op 15 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 15 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36ef952640 5 client.4474.objecter 0 in flight 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e7d1140 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 10 client.4474.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 16 osd.7 2024-04-04T21:03:57.138 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:57.139 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 16 2024-04-04T21:03:57.139 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 15 client.4474.objecter _send_op 16 to 9.1 on osd.7 2024-04-04T21:03:57.139 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:16 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e7d15f0 con 0x559b3e7cd950 2024-04-04T21:03:57.139 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.158 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 5 client.4474.objecter 1 in flight 2024-04-04T21:03:57.158 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e722120 2024-04-04T21:03:57.158 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-04-04T21:03:57.158 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 20 client.4474.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 10 client.4474.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 17 osd.7 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 17 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 15 client.4474.objecter _send_op 17 to 9.1 on osd.7 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:17 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e7225d0 con 0x559b3e7cd950 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 5 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.135+0000 7f36f2653780 5 client.4474.objecter 2 in flight 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (24 ms) 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:57.159 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 16 ondisk uv 4 in 9.1 attempt 0 2024-04-04T21:03:57.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 5 2024-04-04T21:03:57.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 16 2024-04-04T21:03:57.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 15 client.4474.objecter _finish_op 16 2024-04-04T21:03:57.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 5 2024-04-04T21:03:57.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 16 2024-04-04T21:03:57.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 5 client.4474.objecter 1 in flight 2024-04-04T21:03:57.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:57.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-04-04T21:03:57.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:59.581 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 17 ondisk uv 0 in 9.1 attempt 0 2024-04-04T21:03:59.582 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 10 client.4474.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-04T21:03:59.582 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 0 2024-04-04T21:03:59.582 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 17 2024-04-04T21:03:59.582 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 15 client.4474.objecter _finish_op 17 2024-04-04T21:03:59.582 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:59.582 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 17 2024-04-04T21:03:59.582 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36ef952640 5 client.4474.objecter 0 in flight 2024-04-04T21:03:59.582 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36f2653780 20 librados: queue_aio_write 0x559b3e5c8830 completion 0x559b3e7d1030 write_seq 4 2024-04-04T21:03:59.582 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e7d09c0 2024-04-04T21:03:59.582 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:59.582 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36f2653780 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36f2653780 10 client.4474.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 18 osd.7 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 18 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36f2653780 15 client.4474.objecter _send_op 18 to 9.1 on osd.7 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:18 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e7d1140 con 0x559b3e7cd950 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.159+0000 7f36f2653780 5 client.4474.objecter 1 in flight 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.884+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.884+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:57.884+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:58.883+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:58.883+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:03:59.583 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:58.883+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:03:59.584 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:58.997+0000 7f36f23b7640 10 client.4474.objecter tick 2024-04-04T21:03:59.584 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 18 ==== osd_op_reply(18 exist [writefull 0~5] v38'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:03:59.686 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(18 exist [writefull 0~5] v38'5 uv5 ondisk = 0) v8 2024-04-04T21:03:59.686 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:03:59.686 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 18 ondisk uv 5 in 9.1 attempt 0 2024-04-04T21:03:59.686 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 0 2024-04-04T21:03:59.686 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 18 2024-04-04T21:03:59.686 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36ef952640 15 client.4474.objecter _finish_op 18 2024-04-04T21:03:59.686 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:59.686 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 18 2024-04-04T21:03:59.687 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36ef952640 5 client.4474.objecter 0 in flight 2024-04-04T21:03:59.687 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36ef952640 20 librados: complete_aio_write 0x559b3e7d1030 2024-04-04T21:03:59.687 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2423 ms) 2024-04-04T21:03:59.687 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-04-04T21:03:59.687 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36f2653780 20 librados: queue_aio_write 0x559b3e5c8830 completion 0x559b3e814630 write_seq 5 2024-04-04T21:03:59.687 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e7220e0 2024-04-04T21:03:59.687 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:03:59.687 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36f2653780 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:03:59.687 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:03:59.687 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:59.687 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36f2653780 10 client.4474.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 19 osd.7 2024-04-04T21:03:59.687 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:03:59.687 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 19 2024-04-04T21:03:59.687 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36f2653780 15 client.4474.objecter _send_op 19 to 9.1 on osd.7 2024-04-04T21:03:59.688 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:19 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38) v9 -- 0x559b3e722590 con 0x559b3e7cd950 2024-04-04T21:03:59.688 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:03:59.688 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.581+0000 7f36f2653780 5 client.4474.objecter 1 in flight 2024-04-04T21:03:59.688 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.685+0000 7f36ed94e640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 19 ==== osd_map(39..39 src has 1..39) v4 ==== 328+0+0 (crc 0 0 0) 0x7f36e4008d30 con 0x559b3e7cd950 2024-04-04T21:03:59.688 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.685+0000 7f36ed94e640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_map(39..39 src has 1..39) v4 2024-04-04T21:03:59.688 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.685+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map got epochs [39,39] > 38 2024-04-04T21:03:59.688 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.685+0000 7f36ed94e640 3 client.4474.objecter handle_osd_map decoding incremental epoch 39 2024-04-04T21:04:02.507 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.685+0000 7f36ed94e640 10 client.4474.objecter checking op 19 2024-04-04T21:04:02.507 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.685+0000 7f36ed94e640 20 client.4474.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:04:02.507 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.685+0000 7f36ed94e640 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:04:02.507 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.685+0000 7f36ed94e640 20 client.4474.objecter dump_active .. 0 homeless 2024-04-04T21:04:02.507 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.685+0000 7f36ed94e640 20 client.4474.objecter 19 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-04-04T21:04:02.507 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.883+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.883+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:03:59.883+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:00.883+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:00.883+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:00.883+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:01.884+0000 7f36ec94c640 10 monclient: tick 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:01.884+0000 7f36ec94c640 10 monclient: _check_auth_tickets 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:01.884+0000 7f36ec94c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 20 ==== osd_op_reply(19 exist [writefull 0~5] v39'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(19 exist [writefull 0~5] v39'6 uv6 ondisk = 0) v8 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 19 ondisk uv 6 in 9.1 attempt 0 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 0 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 19 2024-04-04T21:04:02.508 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36ef952640 15 client.4474.objecter _finish_op 19 2024-04-04T21:04:02.509 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:04:02.509 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 19 2024-04-04T21:04:02.509 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36ef952640 5 client.4474.objecter 0 in flight 2024-04-04T21:04:02.509 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36ef952640 20 librados: complete_aio_write 0x559b3e814630 2024-04-04T21:04:02.509 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (2925 ms) 2024-04-04T21:04:02.509 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-04-04T21:04:02.509 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36f2653780 20 librados: queue_aio_write 0x559b3e5c8830 completion 0x559b3e722750 write_seq 6 2024-04-04T21:04:02.509 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36f2653780 10 client.4474.objecter _op_submit op 0x559b3e722b10 2024-04-04T21:04:02.509 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36f2653780 20 client.4474.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-04-04T21:04:02.509 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36f2653780 20 client.4474.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-04-04T21:04:02.509 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36f2653780 10 client.4474.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-04-04T21:04:02.804 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36f2653780 20 client.4474.objecter _get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:04:02.804 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36f2653780 10 client.4474.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 20 osd.7 2024-04-04T21:04:02.804 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36f2653780 20 client.4474.objecter get_session s=0x559b3e72db20 osd=7 3 2024-04-04T21:04:02.804 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36f2653780 15 client.4474.objecter _session_op_assign 7 20 2024-04-04T21:04:02.804 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36f2653780 15 client.4474.objecter _send_op 20 to 9.1 on osd.7 2024-04-04T21:04:02.804 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- osd_op(unknown.0.0:20 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) v9 -- 0x559b3e722fc0 con 0x559b3e7cd950 2024-04-04T21:04:02.804 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:04:02.804 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.506+0000 7f36f2653780 5 client.4474.objecter 1 in flight 2024-04-04T21:04:02.804 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36ef952640 1 -- 172.21.15.106:0/1606434848 <== osd.7 v2:172.21.15.203:6808/4104640840 21 ==== osd_op_reply(20 exist [writefull 0~5] v39'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f36e401c040 con 0x559b3e7cd950 2024-04-04T21:04:02.804 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36ef952640 10 client.4474.objecter ms_dispatch 0x559b3e5ffdf0 osd_op_reply(20 exist [writefull 0~5] v39'7 uv7 ondisk = 0) v8 2024-04-04T21:04:02.804 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36ef952640 10 client.4474.objecter in handle_osd_op_reply 2024-04-04T21:04:02.804 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36ef952640 7 client.4474.objecter handle_osd_op_reply 20 ondisk uv 7 in 9.1 attempt 0 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36ef952640 10 client.4474.objecter op 0 rval 0 len 0 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36ef952640 15 client.4474.objecter handle_osd_op_reply completed tid 20 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36ef952640 15 client.4474.objecter _finish_op 20 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36ef952640 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 4 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36ef952640 15 client.4474.objecter _session_op_remove 7 20 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36ef952640 5 client.4474.objecter 0 in flight 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36ef952640 20 librados: complete_aio_write 0x559b3e722750 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (297 ms) 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 10 librados: watch_flush enter 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 10 librados: watch_flush exit 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 10 client.4474.objecter close_session for osd.7 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x559b3e7cd950 msgr2=0x559b3e7cfd40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 1 --2- 172.21.15.106:0/1606434848 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x559b3e7cd950 0x559b3e7cfd40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 20 client.4474.objecter put_session s=0x559b3e72db20 osd=7 1 2024-04-04T21:04:02.805 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 20 client.4474.objecter shutdown clearing up homeless session... 2024-04-04T21:04:02.806 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 10 client.4474.objecter successfully canceled tick 2024-04-04T21:04:02.807 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f36dc06b0d0 msgr2=0x7f36dc06d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:04:02.807 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 1 --2- 172.21.15.106:0/1606434848 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f36dc06b0d0 0x7f36dc06d590 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x559b3e6590b0 tx=0x7f36e000c040 comp rx=0 tx=0).stop 2024-04-04T21:04:02.807 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 10 monclient: shutdown 2024-04-04T21:04:02.807 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:04:02.807 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b3e72a7e0 msgr2=0x559b3e7193b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:04:02.808 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 1 --2- 172.21.15.106:0/1606434848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b3e72a7e0 0x559b3e7193b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f36e800b440 tx=0x7f36e800b900 comp rx=0 tx=0).stop 2024-04-04T21:04:02.808 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f0954640 1 -- 172.21.15.106:0/1606434848 reap_dead start 2024-04-04T21:04:02.808 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 shutdown_connections 2024-04-04T21:04:02.808 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 >> 172.21.15.106:0/1606434848 conn(0x559b3e697930 msgr2=0x559b3e694ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:04:02.808 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 shutdown_connections 2024-04-04T21:04:02.808 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 1 -- 172.21.15.106:0/1606434848 wait complete. 2024-04-04T21:04:02.808 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.803+0000 7f36f2653780 1 librados: shutdown 2024-04-04T21:04:02.808 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [----------] 12 tests from AsioRados (6383 ms total) 2024-04-04T21:04:02.808 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: 2024-04-04T21:04:02.808 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [----------] Global test environment tear-down 2024-04-04T21:04:02.808 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (29012 ms total) 2024-04-04T21:04:02.808 INFO:tasks.workunit.client.0.smithi106.stdout: api_asio: [ PASSED ] 12 tests. 2024-04-04T21:04:05.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 453 creating+peering 2024-04-04T21:04:05.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 125 creating+activating 2024-04-04T21:04:05.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 30 active+clean 2024-04-04T21:04:05.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 2024-04-04T21:04:05.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 2024-04-04T21:04:05.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5628 ms) 2024-04-04T21:04:05.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: [ RUN ] LibRadosService.Status 2024-04-04T21:04:05.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: [ OK ] LibRadosService.Status (20048 ms) 2024-04-04T21:04:05.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: [----------] 4 tests from LibRadosService (31219 ms total) 2024-04-04T21:04:05.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: 2024-04-04T21:04:05.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: [----------] Global test environment tear-down 2024-04-04T21:04:05.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31221 ms total) 2024-04-04T21:04:05.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_service: [ PASSED ] 4 tests. 2024-04-04T21:04:06.054 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-04-04T21:04:06.054 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [----------] Global test environment set-up. 2024-04-04T21:04:06.054 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-04-04T21:04:06.054 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-04-04T21:04:06.054 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5604 ms) 2024-04-04T21:04:06.054 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-04-04T21:04:06.054 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (266 ms) 2024-04-04T21:04:06.054 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-04-04T21:04:06.054 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20988 ms) 2024-04-04T21:04:06.054 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-04-04T21:04:06.054 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: attempt 0 of 20 2024-04-04T21:04:06.055 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5189 ms) 2024-04-04T21:04:06.055 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (32048 ms total) 2024-04-04T21:04:06.055 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: 2024-04-04T21:04:06.055 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [----------] Global test environment tear-down 2024-04-04T21:04:06.055 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (32048 ms total) 2024-04-04T21:04:06.055 INFO:tasks.workunit.client.0.smithi106.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-04-04T21:04:10.219 INFO:tasks.ceph.mon.b.smithi203.stderr:2024-04-04T21:04:10.171+0000 ff99640 -1 mon.b@1(peon).paxos(paxos updating c 1..205) lease_expire from mon.0 v2:172.21.15.106:3300/0 is 0.526487648s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2024-04-04T21:04:13.407 INFO:tasks.workunit.client.0.smithi106.stdout: handler_error: Running main() from gmock_main.cc 2024-04-04T21:04:13.407 INFO:tasks.workunit.client.0.smithi106.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-04-04T21:04:13.407 INFO:tasks.workunit.client.0.smithi106.stdout: handler_error: [----------] Global test environment set-up. 2024-04-04T21:04:13.408 INFO:tasks.workunit.client.0.smithi106.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-04-04T21:04:13.408 INFO:tasks.workunit.client.0.smithi106.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-04-04T21:04:13.408 INFO:tasks.workunit.client.0.smithi106.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (39238 ms) 2024-04-04T21:04:13.408 INFO:tasks.workunit.client.0.smithi106.stdout: handler_error: [----------] 1 test from neocls_handler_error (39238 ms total) 2024-04-04T21:04:13.408 INFO:tasks.workunit.client.0.smithi106.stdout: handler_error: 2024-04-04T21:04:13.408 INFO:tasks.workunit.client.0.smithi106.stdout: handler_error: [----------] Global test environment tear-down 2024-04-04T21:04:13.408 INFO:tasks.workunit.client.0.smithi106.stdout: handler_error: [==========] 1 test from 1 test suite ran. (39238 ms total) 2024-04-04T21:04:13.408 INFO:tasks.workunit.client.0.smithi106.stdout: handler_error: [ PASSED ] 1 test. 2024-04-04T21:04:13.414 INFO:tasks.workunit.client.0.smithi106.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-04-04T21:04:13.414 INFO:tasks.workunit.client.0.smithi106.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-04-04T21:04:13.414 INFO:tasks.workunit.client.0.smithi106.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-04-04T21:04:13.414 INFO:tasks.workunit.client.0.smithi106.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-04-04T21:04:13.414 INFO:tasks.workunit.client.0.smithi106.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (39376 ms) 2024-04-04T21:04:13.414 INFO:tasks.workunit.client.0.smithi106.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (39376 ms total) 2024-04-04T21:04:13.414 INFO:tasks.workunit.client.0.smithi106.stdout: api_cls_remote_reads: 2024-04-04T21:04:13.414 INFO:tasks.workunit.client.0.smithi106.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-04-04T21:04:13.414 INFO:tasks.workunit.client.0.smithi106.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (39376 ms total) 2024-04-04T21:04:13.414 INFO:tasks.workunit.client.0.smithi106.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-04-04T21:04:24.025 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:03:15.542 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:24.037 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:03:15.554 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:24.150 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:03:15.666 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:24.154 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:03:15.670 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:24.305 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:15.787 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:24.311 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:15.793 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:24.386 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:03:15.903 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:24.523 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:03:16.039 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:24.805 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:16.287 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:28.263 INFO:tasks.workunit.client.0.smithi106.stdout: pool: Running main() from gmock_main.cc 2024-04-04T21:04:28.263 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-04-04T21:04:28.263 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [----------] Global test environment set-up. 2024-04-04T21:04:28.263 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-04-04T21:04:28.263 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-04-04T21:04:28.264 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [ OK ] NeoRadosPools.PoolList (6669 ms) 2024-04-04T21:04:28.264 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-04-04T21:04:28.264 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (3971 ms) 2024-04-04T21:04:28.264 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-04-04T21:04:28.264 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2852 ms) 2024-04-04T21:04:28.264 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-04-04T21:04:28.264 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (31496 ms) 2024-04-04T21:04:28.264 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-04-04T21:04:28.264 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (3799 ms) 2024-04-04T21:04:28.264 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-04-04T21:04:28.264 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (5277 ms) 2024-04-04T21:04:28.264 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [----------] 6 tests from NeoRadosPools (54064 ms total) 2024-04-04T21:04:28.264 INFO:tasks.workunit.client.0.smithi106.stdout: pool: 2024-04-04T21:04:28.264 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [----------] Global test environment tear-down 2024-04-04T21:04:28.265 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [==========] 6 tests from 1 test suite ran. (54065 ms total) 2024-04-04T21:04:28.265 INFO:tasks.workunit.client.0.smithi106.stdout: pool: [ PASSED ] 6 tests. 2024-04-04T21:04:30.871 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-04-04T21:04:30.871 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [----------] Global test environment set-up. 2024-04-04T21:04:30.871 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-04-04T21:04:30.871 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-04-04T21:04:30.871 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-04-04T21:04:30.871 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-04-04T21:04:30.871 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: 2024-04-04T21:04:30.871 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-04-04T21:04:30.871 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-04-04T21:04:30.871 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-04T21:04:30.871 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: 2024-04-04T21:03:34.020+0000 7f12d4640780 -1 failed for service _ceph-mon._tcp 2024-04-04T21:04:30.872 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: 2024-04-04T21:03:34.020+0000 7f12d4640780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-04-04T21:04:30.872 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (196 ms) 2024-04-04T21:04:30.872 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-04-04T21:04:30.872 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2023 ms) 2024-04-04T21:04:30.872 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2219 ms total) 2024-04-04T21:04:30.872 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: 2024-04-04T21:04:30.872 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-04-04T21:04:30.872 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-04-04T21:04:30.872 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9aab6e0 2024-04-04T21:04:30.872 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9ab6db0 2024-04-04T21:04:30.872 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b985c600 2024-04-04T21:04:30.872 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9aaa140 2024-04-04T21:04:30.872 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b98607b0 2024-04-04T21:04:30.872 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636ba4a8470 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9b3c730 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be12e0 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be1e20 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be2960 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be1a70 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be1c20 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be3d50 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be2610 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be4890 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be5330 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be3b50 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be6270 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be5030 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be5140 2024-04-04T21:04:30.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be5990 2024-04-04T21:04:30.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be5e80 2024-04-04T21:04:30.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be5fd0 2024-04-04T21:04:30.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be60e0 2024-04-04T21:04:30.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be49c0 2024-04-04T21:04:30.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be4ad0 2024-04-04T21:04:30.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be4be0 2024-04-04T21:04:30.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be4cf0 2024-04-04T21:04:30.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be3050 2024-04-04T21:04:30.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be3610 2024-04-04T21:04:30.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be3720 2024-04-04T21:04:30.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be3830 2024-04-04T21:04:30.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be3160 2024-04-04T21:04:30.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be3290 2024-04-04T21:04:30.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be33a0 2024-04-04T21:04:30.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be6420 2024-04-04T21:04:30.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be54e0 2024-04-04T21:04:30.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be5aa0 2024-04-04T21:04:30.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be5bb0 2024-04-04T21:04:30.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be5cc0 2024-04-04T21:04:30.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be4310 2024-04-04T21:04:30.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be55f0 2024-04-04T21:04:30.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be5700 2024-04-04T21:04:30.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be5850 2024-04-04T21:04:30.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be65b0 2024-04-04T21:04:30.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be6bb0 2024-04-04T21:04:30.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be3e60 2024-04-04T21:04:30.876 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be4420 2024-04-04T21:04:30.876 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be4530 2024-04-04T21:04:30.876 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be4640 2024-04-04T21:04:30.876 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be7680 2024-04-04T21:04:44.320 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:03:35.837 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:44.401 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:03:35.918 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:45.023 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:03:36.540 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:45.403 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:03:36.919 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:45.428 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:03:36.944 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:45.529 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:37.011 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:45.566 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:37.048 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:45.676 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:03:37.192 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:45.725 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:37.207 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:04:53.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be7ca0 2024-04-04T21:04:53.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be82a0 2024-04-04T21:04:53.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be77f0 2024-04-04T21:04:53.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be3f70 2024-04-04T21:04:53.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be40a0 2024-04-04T21:04:53.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be41f0 2024-04-04T21:04:53.107 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be8900 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be7df0 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be7900 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be7a10 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be34d0 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9b3bbf0 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 0x5636b9be7b20 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: started 64 aios 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9aab6e0 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9ab6db0 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b985c600 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9aaa140 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b98607b0 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636ba4a8470 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9b3c730 2024-04-04T21:04:53.108 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be12e0 2024-04-04T21:04:53.109 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be1e20 2024-04-04T21:04:53.109 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be2960 2024-04-04T21:04:53.109 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be1a70 2024-04-04T21:04:53.109 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be1c20 2024-04-04T21:04:53.109 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be3d50 2024-04-04T21:04:53.109 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be2610 2024-04-04T21:04:53.109 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be4890 2024-04-04T21:04:53.109 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be5330 2024-04-04T21:04:53.109 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be3b50 2024-04-04T21:04:53.109 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be6270 2024-04-04T21:04:53.109 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be5030 2024-04-04T21:04:53.109 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be5140 2024-04-04T21:04:53.109 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be5990 2024-04-04T21:04:53.109 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be5e80 2024-04-04T21:04:53.110 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be5fd0 2024-04-04T21:04:53.110 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be60e0 2024-04-04T21:04:53.110 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be49c0 2024-04-04T21:04:53.110 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be4ad0 2024-04-04T21:04:53.110 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be4be0 2024-04-04T21:04:53.110 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be4cf0 2024-04-04T21:04:53.110 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be3050 2024-04-04T21:04:53.110 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be3610 2024-04-04T21:04:53.110 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be3720 2024-04-04T21:04:53.110 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be3830 2024-04-04T21:04:53.110 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be3160 2024-04-04T21:04:53.110 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be3290 2024-04-04T21:04:53.110 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be33a0 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be6420 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be54e0 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be5aa0 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be5bb0 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be5cc0 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be4310 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be55f0 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be5700 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be5850 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be65b0 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be6bb0 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be3e60 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be4420 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be4530 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be4640 2024-04-04T21:04:53.111 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be7680 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be7ca0 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be82a0 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be77f0 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be3f70 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be40a0 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be41f0 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be8900 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be7df0 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be7900 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be7a10 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be34d0 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9b3bbf0 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: waiting 0x5636b9be7b20 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: done. 2024-04-04T21:04:53.112 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (77063 ms) 2024-04-04T21:04:53.113 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (77063 ms total) 2024-04-04T21:04:53.113 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: 2024-04-04T21:04:56.734 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: Running main() from gmock_main.cc 2024-04-04T21:04:56.734 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-04-04T21:04:56.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [----------] Global test environment set-up. 2024-04-04T21:04:56.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-04-04T21:04:56.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: seed 41244 2024-04-04T21:04:56.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-04-04T21:04:56.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-04-04T21:04:56.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-04-04T21:04:56.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (16102 ms) 2024-04-04T21:04:56.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-04-04T21:04:56.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (2005 ms) 2024-04-04T21:04:56.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-04-04T21:04:56.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (545 ms) 2024-04-04T21:04:56.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-04-04T21:04:56.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (3161 ms) 2024-04-04T21:04:56.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-04-04T21:04:56.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2850 ms) 2024-04-04T21:04:56.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-04-04T21:04:56.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (747 ms) 2024-04-04T21:04:56.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-04-04T21:04:56.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (44 ms) 2024-04-04T21:04:56.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-04-04T21:04:56.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (119 ms) 2024-04-04T21:04:56.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-04-04T21:04:56.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (75 ms) 2024-04-04T21:04:56.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-04-04T21:04:56.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (54 ms) 2024-04-04T21:04:56.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-04-04T21:04:56.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (103 ms) 2024-04-04T21:04:56.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (107 ms) 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (145 ms) 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (234 ms) 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (465 ms) 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (2323 ms) 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (791 ms) 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (298 ms) 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (66 ms) 2024-04-04T21:04:56.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (30234 ms total) 2024-04-04T21:04:56.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: 2024-04-04T21:04:56.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-04-04T21:04:56.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-04-04T21:04:56.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (26331 ms) 2024-04-04T21:04:56.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-04-04T21:04:56.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (1533 ms) 2024-04-04T21:04:56.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-04-04T21:04:56.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (126 ms) 2024-04-04T21:04:56.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-04-04T21:04:56.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4646 ms) 2024-04-04T21:04:56.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-04-04T21:04:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 cls: Running main() from gmock_main.cc 2024-04-04T21:04:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-04-04T21:04:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: cls: [----------] Global test environment set-up. 2024-04-04T21:04:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-04-04T21:04:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-04-04T21:04:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: cls: [ OK ] NeoRadosCls.DNE (39300 ms) 2024-04-04T21:04:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-04-04T21:04:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (43704 ms) 2024-04-04T21:04:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: cls: [----------] 2 tests from NeoRadosCls (83004 ms total) 2024-04-04T21:04:57.105 INFO:tasks.workunit.client.0.smithi106.stdout: cls: 2024-04-04T21:04:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: cls: [----------] Global test environment tear-down 2024-04-04T21:04:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: cls: [==========] 2 tests from 1 test suite ran. (83004 ms total) 2024-04-04T21:04:57.106 INFO:tasks.workunit.client.0.smithi106.stdout: cls: [ PASSED ] 2 tests. 2024-04-04T21:04:57.267 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: Running main() from gmock_main.cc 2024-04-04T21:04:57.267 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-04-04T21:04:57.267 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: [----------] Global test environment set-up. 2024-04-04T21:04:57.267 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-04-04T21:04:57.267 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-04-04T21:04:57.267 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (6736 ms) 2024-04-04T21:04:57.267 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-04-04T21:04:57.268 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (26775 ms) 2024-04-04T21:04:57.268 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-04-04T21:04:57.268 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (49655 ms) 2024-04-04T21:04:57.268 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: [----------] 3 tests from NeoRadosCmd (83166 ms total) 2024-04-04T21:04:57.268 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: 2024-04-04T21:04:57.268 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: [----------] Global test environment tear-down 2024-04-04T21:04:57.268 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: [==========] 3 tests from 1 test suite ran. (83166 ms total) 2024-04-04T21:04:57.268 INFO:tasks.workunit.client.0.smithi106.stdout: cmd: [ PASSED ] 3 tests. 2024-04-04T21:04:57.312 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-04-04T21:04:57.312 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-04-04T21:04:57.313 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-04-04T21:04:57.313 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-04-04T21:04:57.313 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-04-04T21:04:57.313 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (984 ms) 2024-04-04T21:04:57.313 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-04-04T21:04:57.313 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (94 ms) 2024-04-04T21:04:57.313 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-04-04T21:04:57.313 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (82288 ms) 2024-04-04T21:04:57.313 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (83366 ms total) 2024-04-04T21:04:57.313 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: 2024-04-04T21:04:57.313 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-04-04T21:04:57.313 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (83366 ms total) 2024-04-04T21:04:57.313 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-04-04T21:04:58.795 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: Running main() from gmock_main.cc 2024-04-04T21:04:58.795 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-04-04T21:04:58.795 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: [----------] Global test environment set-up. 2024-04-04T21:04:58.795 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-04-04T21:04:58.795 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-04-04T21:04:58.795 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: handle_notify cookie 93925939550480 notify_id 167503724544 notifier_gid 4594 2024-04-04T21:04:58.795 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (40435 ms) 2024-04-04T21:04:58.795 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-04-04T21:04:58.795 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: Trying... 2024-04-04T21:04:58.795 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: handle_notify cookie 93925952930528 notify_id 274877906944 notifier_gid 4661 2024-04-04T21:04:58.795 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: Waiting for 3.000000000s 2024-04-04T21:04:58.795 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: Timed out. 2024-04-04T21:04:58.795 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: Flushing... 2024-04-04T21:04:58.796 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: Flushed... 2024-04-04T21:04:58.796 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (44128 ms) 2024-04-04T21:04:58.796 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (84563 ms total) 2024-04-04T21:04:58.796 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: 2024-04-04T21:04:58.796 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: [----------] Global test environment tear-down 2024-04-04T21:04:58.796 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (84563 ms total) 2024-04-04T21:04:58.796 INFO:tasks.workunit.client.0.smithi106.stdout: watch_notify: [ PASSED ] 2 tests. 2024-04-04T21:04:58.797 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-04-04T21:04:58.797 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-04-04T21:04:58.797 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-04-04T21:04:58.797 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-04-04T21:04:58.797 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-04-04T21:04:58.797 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-04-04T21:04:58.797 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-04-04T21:04:58.798 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (21248 ms) 2024-04-04T21:04:58.798 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-04-04T21:04:58.798 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (41109 ms) 2024-04-04T21:04:58.798 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-04-04T21:04:58.798 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (7052 ms) 2024-04-04T21:04:58.798 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-04-04T21:04:58.798 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (1220 ms) 2024-04-04T21:04:58.798 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-04-04T21:04:58.798 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (274 ms) 2024-04-04T21:04:58.798 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-04-04T21:04:58.798 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (297 ms) 2024-04-04T21:04:58.798 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-04-04T21:04:58.798 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (93 ms) 2024-04-04T21:04:58.798 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-04-04T21:04:58.799 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (79 ms) 2024-04-04T21:04:58.799 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-04-04T21:04:58.799 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (88 ms) 2024-04-04T21:04:58.799 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-04-04T21:04:58.799 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (139 ms) 2024-04-04T21:04:58.799 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-04-04T21:04:58.799 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (1944 ms) 2024-04-04T21:04:58.799 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-04-04T21:04:58.799 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (87 ms) 2024-04-04T21:04:58.799 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-04-04T21:04:58.799 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (703 ms) 2024-04-04T21:04:58.799 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-04-04T21:04:58.799 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (355 ms) 2024-04-04T21:04:58.799 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-04-04T21:04:58.800 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (223 ms) 2024-04-04T21:04:58.800 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-04-04T21:04:58.800 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (66 ms) 2024-04-04T21:04:58.800 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (74978 ms total) 2024-04-04T21:04:58.800 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: 2024-04-04T21:04:58.800 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-04-04T21:04:58.800 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (84795 ms total) 2024-04-04T21:04:58.800 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-04-04T21:04:58.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: Running main() from gmock_main.cc 2024-04-04T21:04:58.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-04-04T21:04:58.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [----------] Global test environment set-up. 2024-04-04T21:04:58.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-04-04T21:04:58.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-04-04T21:04:58.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (986 ms) 2024-04-04T21:04:58.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-04-04T21:04:58.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (85 ms) 2024-04-04T21:04:58.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-04-04T21:04:58.873 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (82121 ms) 2024-04-04T21:04:58.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-04-04T21:04:58.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:46.634289+0000 mon.b [INF] from='client.4906 172.21.15.106:0/348609216' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi106-41516-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-04T21:04:58.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:47.345230+0000 mon.a [INF] from='client.4906 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi106-41516-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-04T21:04:58.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:48.518854+0000 mon.b [INF] from='client.? 172.21.15.106:0/319768984' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi106-41404-4"}]: dispatch 2024-04-04T21:04:58.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:52.362545+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi106-41404-4"}]: dispatch 2024-04-04T21:04:58.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:52.536287+0000 mon.a [INF] from='client.4906 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi106-41516-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-04T21:04:58.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:54.853873+0000 mon.a [WRN] Health check update: 24 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-04T21:04:58.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:54.979071+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on front (longest 6874.040ms) (OSD_SLOW_PING_TIME_FRONT) 2024-04-04T21:04:58.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:56.199912+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi106-41404-4"}]': finished 2024-04-04T21:04:58.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:56.819443+0000 mon.a [INF] from='client.? 172.21.15.106:0/679866632' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi106-41594-7"}]: dispatch 2024-04-04T21:04:58.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:57.151158+0000 client.admin [INF] onexx 2024-04-04T21:04:58.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:57.176687+0000 mon.a [INF] from='client.? 172.21.15.106:0/3067912998' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-04T21:04:58.874 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:57.205797+0000 mon.b [INF] from='client.? 172.21.15.106:0/319768984' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi106-41404-4", "tierpool": "test-rados-api-smithi106-41404-4-cache"}]: dispatch 2024-04-04T21:04:58.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:58.032459+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi106-41404-4", "tierpool": "test-rados-api-smithi106-41404-4-cache"}]: dispatch 2024-04-04T21:04:58.875 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:58.069487+0000 mon.a [INF] from='client.? 172.21.15.106:0/679866632' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi106-41594-7"}]': finished 2024-04-04T21:04:59.756 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:58.445873+0000 mon.b [INF] from='client.? 172.21.15.106:0/1652893618' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-ap api_stat_pp: Running main() from gmock_main.cc 2024-04-04T21:04:59.756 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-04-04T21:04:59.756 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-04-04T21:04:59.756 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-04-04T21:04:59.757 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: seed 41594 2024-04-04T21:04:59.757 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-04-04T21:04:59.757 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (23898 ms) 2024-04-04T21:04:59.757 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-04-04T21:04:59.757 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (143 ms) 2024-04-04T21:04:59.757 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-04-04T21:04:59.757 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1484 ms) 2024-04-04T21:04:59.757 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-04-04T21:04:59.757 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (6482 ms) 2024-04-04T21:04:59.757 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-04-04T21:04:59.757 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (121 ms) 2024-04-04T21:04:59.757 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (32128 ms total) 2024-04-04T21:04:59.757 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: 2024-04-04T21:04:59.757 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-04-04T21:04:59.758 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-04-04T21:04:59.758 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (26002 ms) 2024-04-04T21:04:59.758 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-04-04T21:04:59.758 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (298 ms) 2024-04-04T21:04:59.758 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-04-04T21:04:59.758 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1211 ms) 2024-04-04T21:04:59.758 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-04-04T21:04:59.758 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (573 ms) 2024-04-04T21:04:59.758 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (28084 ms total) 2024-04-04T21:04:59.758 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: 2024-04-04T21:04:59.758 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-04-04T21:04:59.758 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (85875 ms total) 2024-04-04T21:04:59.758 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-04-04T21:04:59.808 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_1_[42111]: starting. 2024-04-04T21:04:59.808 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_1_[42111]: creating pool ceph_test_rados_open_pools_parallel.smithi106-42055 2024-04-04T21:04:59.808 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_1_[42111]: created object 0... 2024-04-04T21:04:59.808 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_1_[42111]: created object 25... 2024-04-04T21:04:59.808 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_1_[42111]: created object 49... 2024-04-04T21:04:59.808 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_1_[42111]: finishing. 2024-04-04T21:04:59.808 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_1_[42111]: shutting down. 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_2_[42114]: starting. 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_2_[42114]: rados_pool_create. 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_2_[42114]: rados_ioctx_create. 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_2_[42114]: shutting down. 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: ******************************* 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_3_[43537]: starting. 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_3_[43537]: creating pool ceph_test_rados_open_pools_parallel.smithi106-42055 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_3_[43537]: created object 0... 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_3_[43537]: created object 25... 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_3_[43537]: created object 49... 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_3_[43537]: finishing. 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_3_[43537]: shutting down. 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: ******************************* 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_4_[43538]: starting. 2024-04-04T21:04:59.809 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_4_[43538]: rados_pool_create. 2024-04-04T21:04:59.810 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_4_[43538]: rados_ioctx_create. 2024-04-04T21:04:59.810 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: process_4_[43538]: shutting down. 2024-04-04T21:04:59.810 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: ******************************* 2024-04-04T21:04:59.810 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: ******************************* 2024-04-04T21:04:59.810 INFO:tasks.workunit.client.0.smithi106.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-04-04T21:05:07.990 INFO:tasks.workunit.client.0.smithi106.stdout:i-smithi106-41893-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-04T21:05:07.991 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:04:58.552382+0000 mon.a [INF] from='client.? 172.21.15.106:0/679866632' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi106-41594-7"}]: dispatch 2024-04-04T21:05:07.991 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:05:00.988499+0000 mon.a [INF] from='client.? 172.21.15.106:0/3067912998' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-04T21:05:07.991 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:05:01.381741+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi106-41566-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi106-41566-12"}]': finished 2024-04-04T21:05:07.991 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:05:01.647337+0000 mon.b [INF] from='client.? 172.21.15.106:0/3959777587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi106-41516-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-04T21:05:07.991 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:05:01.663670+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi106-41516-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-04T21:05:07.991 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:05:02.459561+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi106-41516-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-04T21:05:07.991 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:05:03.000969+0000 client.admin [INF] threexx 2024-04-04T21:05:07.991 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: got: 2024-04-04T21:05:03.004779+0000 mon.a [INF] from='client.? 172.21.15.106:0/3067912998' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-04T21:05:07.991 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (10883 ms) 2024-04-04T21:05:07.991 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (94075 ms total) 2024-04-04T21:05:07.991 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: 2024-04-04T21:05:07.992 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [----------] Global test environment tear-down 2024-04-04T21:05:07.992 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (94075 ms total) 2024-04-04T21:05:07.992 INFO:tasks.workunit.client.0.smithi106.stdout: api_cmd: [ PASSED ] 4 tests. 2024-04-04T21:05:10.425 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: Running main() from gmock_main.cc 2024-04-04T21:05:10.426 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-04-04T21:05:10.426 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [----------] Global test environment set-up. 2024-04-04T21:05:10.426 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-04-04T21:05:10.426 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-04-04T21:05:10.426 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ OK ] LibRadosPools.PoolList (9574 ms) 2024-04-04T21:05:10.426 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-04-04T21:05:10.426 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (4284 ms) 2024-04-04T21:05:10.426 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-04-04T21:05:10.426 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (30269 ms) 2024-04-04T21:05:10.426 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-04-04T21:05:10.426 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (5022 ms) 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (7388 ms) 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6525 ms) 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (6026 ms) 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (17581 ms) 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (9917 ms) 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [----------] 9 tests from LibRadosPools (96587 ms total) 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [----------] Global test environment tear-down 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (96587 ms total) 2024-04-04T21:05:10.427 INFO:tasks.workunit.client.0.smithi106.stdout: api_pool: [ PASSED ] 9 tests. 2024-04-04T21:05:16.094 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:04:07.611 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:16.113 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:04:07.629 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:16.115 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:07.598 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:16.117 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:07.600 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:16.118 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:04:07.635 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:16.121 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:04:07.638 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:16.331 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:04:07.847 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:16.355 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:07.837 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:16.375 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:04:07.892 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:31.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: Running main() from gmock_main.cc 2024-04-04T21:05:31.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-04-04T21:05:31.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [----------] Global test environment set-up. 2024-04-04T21:05:31.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-04-04T21:05:31.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-04-04T21:05:31.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ OK ] LibRadosStat.Stat (21247 ms) 2024-04-04T21:05:31.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-04-04T21:05:31.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (3929 ms) 2024-04-04T21:05:31.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-04-04T21:05:31.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ OK ] LibRadosStat.StatNS (632 ms) 2024-04-04T21:05:31.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-04-04T21:05:31.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (4506 ms) 2024-04-04T21:05:31.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-04-04T21:05:31.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (432 ms) 2024-04-04T21:05:31.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [----------] 5 tests from LibRadosStat (30746 ms total) 2024-04-04T21:05:31.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: 2024-04-04T21:05:31.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-04-04T21:05:31.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-04-04T21:05:31.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (44668 ms) 2024-04-04T21:05:31.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-04-04T21:05:31.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (18343 ms) 2024-04-04T21:05:31.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-04-04T21:05:31.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (517 ms) 2024-04-04T21:05:31.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-04-04T21:05:31.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (976 ms) 2024-04-04T21:05:31.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (64504 ms total) 2024-04-04T21:05:31.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: 2024-04-04T21:05:31.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [----------] Global test environment tear-down 2024-04-04T21:05:31.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (117847 ms total) 2024-04-04T21:05:31.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_stat: [ PASSED ] 9 tests. 2024-04-04T21:05:31.727 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-04-04T21:05:31.727 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-04-04T21:05:31.727 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-04-04T21:05:31.727 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-04-04T21:05:31.727 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-04-04T21:05:31.727 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: watch_notify_test_cb 2024-04-04T21:05:31.727 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (25286 ms) 2024-04-04T21:05:31.727 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-04-04T21:05:31.727 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-04-04T21:05:31.727 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94391339595392 err -107 2024-04-04T21:05:31.727 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1501 ms) 2024-04-04T21:05:31.727 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94391339597520 err -107 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (3011 ms) 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: watch_notify2_test_cb from 4430 notify_id 171798691841 cookie 94391339600784 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (548 ms) 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: watch_notify2_test_cb from 4430 notify_id 171798691841 cookie 94391339600704 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (120 ms) 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: watch_notify2_test_cb from 4430 notify_id 171798691840 cookie 94391339652608 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (223 ms) 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: watch_notify2_test_cb from 4430 notify_id 171798691840 cookie 94391339635808 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: watch_notify2_test_cb from 4430 notify_id 171798691840 cookie 94391339665376 2024-04-04T21:05:31.728 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (307 ms) 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: watch_notify2_test_cb from 4430 notify_id 171798691841 cookie 94391339635808 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: watch_notify2_test_cb from 4430 notify_id 171798691842 cookie 94391339635808 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (8631 ms) 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94391339635680 err -107 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: watch_notify2_test_cb from 4430 notify_id 274877906946 cookie 94391339635680 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (34506 ms) 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94391339635680 err -107 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1784 ms) 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (75918 ms total) 2024-04-04T21:05:31.729 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: 2024-04-04T21:05:31.730 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-04-04T21:05:31.730 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-04-04T21:05:31.730 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: watch_notify_test_cb 2024-04-04T21:05:31.730 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (26600 ms) 2024-04-04T21:05:31.730 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (26600 ms total) 2024-04-04T21:05:31.730 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: 2024-04-04T21:05:31.730 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-04-04T21:05:31.730 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (117871 ms total) 2024-04-04T21:05:31.730 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-04-04T21:05:35.834 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:04:27.351 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:35.841 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:04:27.358 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:35.896 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:27.378 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:35.902 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:27.384 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:35.908 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:04:27.425 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:35.912 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:04:27.428 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:36.026 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:04:27.543 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:36.036 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:27.518 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:36.067 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:04:27.584 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:05:40.385 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-04T21:05:40.356+0000 ea52640 -1 osd.5 104 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4766.0:3 97.f 97.d23e646f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-04-04T21:05:41.374 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-04T21:05:41.370+0000 ea52640 -1 osd.5 104 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4766.0:3 97.f 97.d23e646f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-04-04T21:05:43.769 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-04T21:05:43.651+0000 ea52640 -1 osd.5 106 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4766.0:3 97.f 97.d23e646f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-04-04T21:05:44.706 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-04T21:05:44.698+0000 ea52640 -1 osd.5 106 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4766.0:3 97.f 97.d23e646f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-04-04T21:05:45.727 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-04T21:05:45.723+0000 ea52640 -1 osd.5 106 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4766.0:3 97.f 97:f6267c4b:test-rados-api-smithi106-41404-8::foo:head [writefull 0~4 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-04-04T21:05:51.144 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-04-04T21:05:51.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [----------] Global test environment set-up. 2024-04-04T21:05:51.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [----------] 7 tests from LibRadosList 2024-04-04T21:05:51.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-04-04T21:05:51.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ OK ] LibRadosList.ListObjects (26533 ms) 2024-04-04T21:05:51.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-04-04T21:05:51.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (5026 ms) 2024-04-04T21:05:51.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-04-04T21:05:51.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: myset foo1,foo2,foo3 2024-04-04T21:05:51.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo1 2024-04-04T21:05:51.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo2 2024-04-04T21:05:51.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo3 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: myset foo1,foo4,foo5 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo4 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo5 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo1 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: myset foo6,foo7 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo7 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo6 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: ns1:foo4 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: ns1:foo5 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: ns2:foo7 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: ns2:foo6 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: ns1:foo1 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: :foo1 2024-04-04T21:05:51.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: :foo2 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: :foo3 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (1883 ms) 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 1 0 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 10 0 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 13 0 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 7 0 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 14 0 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 0 0 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 15 0 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 11 0 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 5 0 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 8 0 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 6 0 2024-04-04T21:05:51.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 3 0 2024-04-04T21:05:51.207 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 4 0 2024-04-04T21:05:51.207 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 12 0 2024-04-04T21:05:51.207 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 9 0 2024-04-04T21:05:51.207 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 2 0 2024-04-04T21:05:51.207 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-04-04T21:05:51.207 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (44280 ms) 2024-04-04T21:05:51.207 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-04-04T21:05:51.207 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: x cursor=MIN 2024-04-04T21:05:51.207 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=1 cursor=8:02547ec2:::1:head 2024-04-04T21:05:51.207 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=10 cursor=8:52ea6a34:::10:head 2024-04-04T21:05:51.207 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=13 cursor=8:566253c9:::13:head 2024-04-04T21:05:51.207 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=7 cursor=8:5c6b0b28:::7:head 2024-04-04T21:05:51.207 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=14 cursor=8:62a1935d:::14:head 2024-04-04T21:05:51.207 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=0 cursor=8:6cac518f:::0:head 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=15 cursor=8:863748b0:::15:head 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=11 cursor=8:89d3ae78:::11:head 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=5 cursor=8:b29083e3:::5:head 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=8 cursor=8:bd63b0f1:::8:head 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=6 cursor=8:c4fdafeb:::6:head 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=3 cursor=8:cfc208b3:::3:head 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=4 cursor=8:d83876eb:::4:head 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=12 cursor=8:de5d7c5f:::12:head 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=9 cursor=8:e960b815:::9:head 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > oid=2 cursor=8:f905c69b:::2:head 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: FIRST> seek to MIN oid=1 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=1 cursor=8:02547ec2:::1:head 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-04-04T21:05:51.208 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=10 cursor=8:52ea6a34:::10:head 2024-04-04T21:05:51.209 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-04-04T21:05:51.209 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-04-04T21:05:51.209 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=13 cursor=8:566253c9:::13:head 2024-04-04T21:05:51.209 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:566253c9:::13:head 2024-04-04T21:05:51.209 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-04-04T21:05:51.209 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=7 cursor=8:5c6b0b28:::7:head 2024-04-04T21:05:51.209 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-04-04T21:05:51.209 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-04-04T21:05:52.106 INFO:tasks.ceph.mon.c.smithi106.stderr:2024-04-04T21:05:52.059+0000 ff99640 -1 mon.c@2(peon).paxos(paxos updating c 1..350) lease_expire from mon.0 v2:172.21.15.106:3300/0 is 1.179579854s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2024-04-04T21:05:53.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=14 cursor=8:62a1935d:::14:head 2024-04-04T21:05:53.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-04-04T21:05:53.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-04-04T21:05:53.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=0 cursor=8:6cac518f:::0:head 2024-04-04T21:05:53.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-04-04T21:05:53.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-04-04T21:05:53.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=15 cursor=8:863748b0:::15:head 2024-04-04T21:05:53.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:863748b0:::15:head 2024-04-04T21:05:53.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-04-04T21:05:53.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=11 cursor=8:89d3ae78:::11:head 2024-04-04T21:05:53.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=5 cursor=8:b29083e3:::5:head 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=8 cursor=8:bd63b0f1:::8:head 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=6 cursor=8:c4fdafeb:::6:head 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=3 cursor=8:cfc208b3:::3:head 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=4 cursor=8:d83876eb:::4:head 2024-04-04T21:05:53.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-04-04T21:05:53.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-04-04T21:05:53.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=12 cursor=8:de5d7c5f:::12:head 2024-04-04T21:05:53.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-04-04T21:05:53.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-04-04T21:05:53.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=9 cursor=8:e960b815:::9:head 2024-04-04T21:05:53.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:e960b815:::9:head 2024-04-04T21:05:53.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-04-04T21:05:53.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : oid=2 cursor=8:f905c69b:::2:head 2024-04-04T21:05:53.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-04-04T21:05:53.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-04-04T21:05:53.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-04-04T21:05:53.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:6cac518f:::0:head expected=8:6cac518f:::0:head 2024-04-04T21:05:53.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-04-04T21:05:53.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=0 expected=0 2024-04-04T21:05:53.741 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-04-04T21:05:53.741 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:89d3ae78:::11:head expected=8:89d3ae78:::11:head 2024-04-04T21:05:53.741 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-04-04T21:05:53.741 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=11 expected=11 2024-04-04T21:05:53.741 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:863748b0:::15:head 2024-04-04T21:05:53.741 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:863748b0:::15:head expected=8:863748b0:::15:head 2024-04-04T21:05:53.741 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-04-04T21:05:53.741 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=15 expected=15 2024-04-04T21:05:53.741 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-04-04T21:05:53.741 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:62a1935d:::14:head expected=8:62a1935d:::14:head 2024-04-04T21:05:53.741 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-04-04T21:05:53.741 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=14 expected=14 2024-04-04T21:05:53.742 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:566253c9:::13:head 2024-04-04T21:05:53.742 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:566253c9:::13:head expected=8:566253c9:::13:head 2024-04-04T21:05:53.742 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-04-04T21:05:53.742 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=13 expected=13 2024-04-04T21:05:53.742 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-04-04T21:05:53.742 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:5c6b0b28:::7:head expected=8:5c6b0b28:::7:head 2024-04-04T21:05:53.742 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-04-04T21:05:53.742 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=7 expected=7 2024-04-04T21:05:53.742 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-04-04T21:05:53.742 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:b29083e3:::5:head expected=8:b29083e3:::5:head 2024-04-04T21:05:53.742 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-04-04T21:05:53.742 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=5 expected=5 2024-04-04T21:05:53.742 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-04-04T21:05:53.743 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:52ea6a34:::10:head expected=8:52ea6a34:::10:head 2024-04-04T21:05:53.743 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-04-04T21:05:53.743 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=10 expected=10 2024-04-04T21:06:01.902 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-04-04T21:06:01.893+0000 ea52640 -1 osd.6 107 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4784.0:2 100.15 100.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e90) 2024-04-04T21:06:02.936 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-04-04T21:06:02.934+0000 ea52640 -1 osd.6 107 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4784.0:2 100.15 100.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e90) 2024-04-04T21:06:03.803 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:f905c69b:::2:hea list_parallel: process_1_[41940]: starting. 2024-04-04T21:06:03.803 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_1_[41940]: creating pool ceph_test_rados_list_parallel.smithi106-41915 2024-04-04T21:06:03.803 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_1_[41940]: created object 0... 2024-04-04T21:06:03.803 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_1_[41940]: created object 25... 2024-04-04T21:06:03.803 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_1_[41940]: created object 49... 2024-04-04T21:06:03.803 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_1_[41940]: finishing. 2024-04-04T21:06:03.803 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_1_[41940]: shutting down. 2024-04-04T21:06:03.804 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_2_[41941]: starting. 2024-04-04T21:06:03.804 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_2_[41941]: listing objects. 2024-04-04T21:06:03.804 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_2_[41941]: listed object 0... 2024-04-04T21:06:03.804 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_2_[41941]: listed object 25... 2024-04-04T21:06:03.804 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_2_[41941]: saw 50 objects 2024-04-04T21:06:03.804 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_2_[41941]: shutting down. 2024-04-04T21:06:03.804 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:03.804 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_3_[43810]: starting. 2024-04-04T21:06:03.804 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_3_[43810]: creating pool ceph_test_rados_list_parallel.smithi106-41915 2024-04-04T21:06:03.804 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_3_[43810]: created object 0... 2024-04-04T21:06:03.804 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_3_[43810]: created object 25... 2024-04-04T21:06:03.804 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_3_[43810]: created object 49... 2024-04-04T21:06:03.804 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_3_[43810]: finishing. 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_3_[43810]: shutting down. 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_4_[43811]: starting. 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_4_[43811]: listing objects. 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_4_[43811]: listed object 0... 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_4_[43811]: listed object 25... 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_4_[43811]: saw 33 objects 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_4_[43811]: shutting down. 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_5_[43812]: starting. 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_5_[43812]: removed 25 objects... 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_5_[43812]: removed half of the objects 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_5_[43812]: removed 50 objects... 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_5_[43812]: removed 50 objects 2024-04-04T21:06:03.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_5_[43812]: shutting down. 2024-04-04T21:06:03.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:03.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:03.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_6_[53419]: starting. 2024-04-04T21:06:03.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_6_[53419]: creating pool ceph_test_rados_list_parallel.smithi106-41915 2024-04-04T21:06:03.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_6_[53419]: created object 0... 2024-04-04T21:06:03.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_6_[53419]: created object 25... 2024-04-04T21:06:03.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_6_[53419]: created object 49... 2024-04-04T21:06:03.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_6_[53419]: finishing. 2024-04-04T21:06:03.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_6_[53419]: shutting down. 2024-04-04T21:06:03.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:03.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:03.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_7_[53420]: starting. 2024-04-04T21:06:03.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_7_[53420]: listing objects. 2024-04-04T21:06:03.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_7_[53420]: listed object 0... 2024-04-04T21:06:03.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_7_[53420]: listed object 25... 2024-04-04T21:06:03.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_7_[53420]: listed object 50... 2024-04-04T21:06:03.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_7_[53420]: saw 57 objects 2024-04-04T21:06:03.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_7_[53420]: shutting down. 2024-04-04T21:06:03.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:03.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:03.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_8_[53421]: starting. 2024-04-04T21:06:03.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_8_[53421]: added 25 objects... 2024-04-04T21:06:03.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_8_[53421]: added half of the objects 2024-04-04T21:06:03.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_8_[53421]: added 50 objects... 2024-04-04T21:06:03.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_8_[53421]: added 50 objects 2024-04-04T21:06:03.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_8_[53421]: shutting down. 2024-04-04T21:06:03.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:03.808 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:03.990 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-04-04T21:06:03.985+0000 ea52640 -1 osd.6 107 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4784.0:2 100.15 100.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e90) 2024-04-04T21:06:04.967 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-04-04T21:06:04.962+0000 ea52640 -1 osd.6 107 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4784.0:2 100.15 100.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e90) 2024-04-04T21:06:05.991 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-04-04T21:06:05.988+0000 ea52640 -1 osd.6 107 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4784.0:2 100.15 100.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e90) 2024-04-04T21:06:07.129 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-04-04T21:06:07.011+0000 ea52640 -1 osd.6 108 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4784.0:2 100.15 100.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e90) 2024-04-04T21:06:10.571 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ********************** delete_pools_parallel: process_1_[42124]: starting. 2024-04-04T21:06:10.571 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_1_[42124]: creating pool ceph_test_rados_delete_pools_parallel.smithi106-42095 2024-04-04T21:06:10.571 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_1_[42124]: created object 0... 2024-04-04T21:06:10.571 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_1_[42124]: created object 25... 2024-04-04T21:06:10.571 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_1_[42124]: created object 49... 2024-04-04T21:06:10.571 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_1_[42124]: finishing. 2024-04-04T21:06:10.571 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_1_[42124]: shutting down. 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_2_[42125]: starting. 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_2_[42125]: deleting pool ceph_test_rados_delete_pools_parallel.smithi106-42095 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_2_[42125]: shutting down. 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: ******************************* 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_3_[43470]: starting. 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_3_[43470]: creating pool ceph_test_rados_delete_pools_parallel.smithi106-42095 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_3_[43470]: created object 0... 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_3_[43470]: created object 25... 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_3_[43470]: created object 49... 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_3_[43470]: finishing. 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_3_[43470]: shutting down. 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: ******************************* 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_5_[43472]: starting. 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_5_[43472]: listing objects. 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_5_[43472]: listed object 0... 2024-04-04T21:06:10.572 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_5_[43472]: listed object 25... 2024-04-04T21:06:10.573 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_5_[43472]: saw 50 objects 2024-04-04T21:06:10.573 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_5_[43472]: shutting down. 2024-04-04T21:06:10.573 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: ******************************* 2024-04-04T21:06:10.573 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_4_[43471]: starting. 2024-04-04T21:06:10.573 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_4_[43471]: deleting pool ceph_test_rados_delete_pools_parallel.smithi106-42095 2024-04-04T21:06:10.573 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: process_4_[43471]: shutting down. 2024-04-04T21:06:10.573 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: ******************************* 2024-04-04T21:06:10.573 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: ******************************* 2024-04-04T21:06:10.573 INFO:tasks.workunit.client.0.smithi106.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-04-04T21:06:19.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-04-04T21:06:19.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-04-04T21:06:19.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-04-04T21:06:19.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-04-04T21:06:19.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: seed 41305 2024-04-04T21:06:19.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-04-04T21:06:19.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (10196 ms) 2024-04-04T21:06:19.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-04-04T21:06:19.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (7452 ms) 2024-04-04T21:06:19.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-04-04T21:06:19.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (8728 ms) 2024-04-04T21:06:19.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-04-04T21:06:19.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (3075 ms) 2024-04-04T21:06:19.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-04-04T21:06:19.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (1120 ms) 2024-04-04T21:06:19.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-04-04T21:06:19.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (960 ms) 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (210 ms) 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (80 ms) 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (31821 ms total) 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (27590 ms) 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (319 ms) 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (20789 ms) 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-04-04T21:06:19.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (5409 ms) 2024-04-04T21:06:19.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-04-04T21:06:19.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (13818 ms) 2024-04-04T21:06:19.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-04-04T21:06:19.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (15512 ms) 2024-04-04T21:06:19.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-04-04T21:06:19.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (198 ms) 2024-04-04T21:06:19.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-04-04T21:06:19.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (150 ms) 2024-04-04T21:06:19.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (83785 ms total) 2024-04-04T21:06:19.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: 2024-04-04T21:06:19.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-04-04T21:06:19.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (165980 ms total) 2024-04-04T21:06:19.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-04-04T21:06:20.491 INFO:tasks.workunit.client.0.smithi106.stdout:********* 2024-04-04T21:06:20.491 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_9_[60545]: starting. 2024-04-04T21:06:20.491 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_9_[60545]: creating pool ceph_test_rados_list_parallel.smithi106-41915 2024-04-04T21:06:20.492 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_9_[60545]: created object 0... 2024-04-04T21:06:20.492 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_9_[60545]: created object 25... 2024-04-04T21:06:20.492 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_9_[60545]: created object 49... 2024-04-04T21:06:20.492 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_9_[60545]: finishing. 2024-04-04T21:06:20.492 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_9_[60545]: shutting down. 2024-04-04T21:06:20.492 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.492 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.492 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.492 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_10_[60546]: starting. 2024-04-04T21:06:20.492 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_10_[60546]: listing objects. 2024-04-04T21:06:20.493 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_10_[60546]: listed object 0... 2024-04-04T21:06:20.493 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_10_[60546]: listed object 25... 2024-04-04T21:06:20.493 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_10_[60546]: listed object 50... 2024-04-04T21:06:20.493 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_10_[60546]: listed object 75... 2024-04-04T21:06:20.493 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_10_[60546]: saw 100 objects 2024-04-04T21:06:20.493 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_10_[60546]: shutting down. 2024-04-04T21:06:20.493 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.493 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.493 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.493 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_12_[60548]: starting. 2024-04-04T21:06:20.493 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_12_[60548]: added 25 objects... 2024-04-04T21:06:20.493 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_12_[60548]: added half of the objects 2024-04-04T21:06:20.493 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_12_[60548]: added 50 objects... 2024-04-04T21:06:20.493 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_12_[60548]: added 50 objects 2024-04-04T21:06:20.494 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_12_[60548]: shutting down. 2024-04-04T21:06:20.494 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.494 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.494 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.494 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_13_[60549]: starting. 2024-04-04T21:06:20.494 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_13_[60549]: removed 25 objects... 2024-04-04T21:06:20.494 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_13_[60549]: removed half of the objects 2024-04-04T21:06:20.494 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_13_[60549]: removed 50 objects... 2024-04-04T21:06:20.494 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_13_[60549]: removed 50 objects 2024-04-04T21:06:20.494 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_13_[60549]: shutting down. 2024-04-04T21:06:20.494 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.494 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.495 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.495 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_11_[60547]: starting. 2024-04-04T21:06:20.495 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_11_[60547]: added 25 objects... 2024-04-04T21:06:20.495 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_11_[60547]: added half of the objects 2024-04-04T21:06:20.495 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_11_[60547]: added 50 objects... 2024-04-04T21:06:20.495 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_11_[60547]: added 50 objects 2024-04-04T21:06:20.495 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_11_[60547]: shutting down. 2024-04-04T21:06:20.495 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.495 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.495 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.495 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.495 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_14_[72101]: starting. 2024-04-04T21:06:20.495 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_14_[72101]: creating pool ceph_test_rados_list_parallel.smithi106-41915 2024-04-04T21:06:20.495 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_14_[72101]: created object 0... 2024-04-04T21:06:20.496 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_14_[72101]: created object 25... 2024-04-04T21:06:20.496 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_14_[72101]: created object 49... 2024-04-04T21:06:20.496 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_14_[72101]: finishing. 2024-04-04T21:06:20.496 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_14_[72101]: shutting down. 2024-04-04T21:06:20.496 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.496 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.496 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.496 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:20.496 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_15_[72102]: starting. 2024-04-04T21:06:20.496 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_15_[72102]: listing objects. 2024-04-04T21:06:20.496 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_15_[72102]: listed object 0... 2024-04-04T21:06:22.521 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_15_[72102]: listed object (271 ms) 2024-04-04T21:06:22.521 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-04-04T21:06:22.521 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (19314 ms) 2024-04-04T21:06:22.521 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-04-04T21:06:22.521 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (6406 ms) 2024-04-04T21:06:22.521 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-04-04T21:06:22.521 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (3922 ms) 2024-04-04T21:06:22.521 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-04-04T21:06:22.521 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (8056 ms) 2024-04-04T21:06:22.522 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-04-04T21:06:22.522 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (12369 ms) 2024-04-04T21:06:22.522 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-04-04T21:06:22.522 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (483 ms) 2024-04-04T21:06:22.522 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-04-04T21:06:22.522 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (135 ms) 2024-04-04T21:06:22.522 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-04-04T21:06:22.522 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (2794 ms) 2024-04-04T21:06:22.522 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-04-04T21:06:22.522 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (303 ms) 2024-04-04T21:06:22.522 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-04-04T21:06:22.522 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (334 ms) 2024-04-04T21:06:22.522 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-04-04T21:06:22.522 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (101 ms) 2024-04-04T21:06:22.523 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-04-04T21:06:22.523 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (256 ms) 2024-04-04T21:06:22.523 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (87381 ms total) 2024-04-04T21:06:22.523 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: 2024-04-04T21:06:22.523 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [----------] Global test environment tear-down 2024-04-04T21:06:22.523 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (168793 ms total) 2024-04-04T21:06:22.523 INFO:tasks.workunit.client.0.smithi106.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-04-04T21:06:29.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: Running main() from gmock_main.cc 2024-04-04T21:06:29.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-04-04T21:06:29.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [----------] Global test environment set-up. 2024-04-04T21:06:29.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-04-04T21:06:29.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-04-04T21:06:29.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (14897 ms) 2024-04-04T21:06:29.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-04-04T21:06:29.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLock.LockShared (3443 ms) 2024-04-04T21:06:29.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-04-04T21:06:29.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (5637 ms) 2024-04-04T21:06:29.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-04-04T21:06:29.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1802 ms) 2024-04-04T21:06:29.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-04-04T21:06:29.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (429 ms) 2024-04-04T21:06:29.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-04-04T21:06:29.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLock.Unlock (220 ms) 2024-04-04T21:06:29.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-04-04T21:06:29.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (157 ms) 2024-04-04T21:06:29.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-04-04T21:06:29.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (100 ms) 2024-04-04T21:06:29.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [----------] 8 tests from LibRadosLock (26685 ms total) 2024-04-04T21:06:29.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: 2024-04-04T21:06:29.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-04-04T21:06:29.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-04-04T21:06:29.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (24350 ms) 2024-04-04T21:06:29.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-04-04T21:06:29.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (1787 ms) 2024-04-04T21:06:29.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-04-04T21:06:29.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (20628 ms) 2024-04-04T21:06:29.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-04-04T21:06:29.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (5668 ms) 2024-04-04T21:06:29.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-04-04T21:06:29.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (16278 ms) 2024-04-04T21:06:29.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-04-04T21:06:29.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (3841 ms) 2024-04-04T21:06:29.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-04-04T21:06:29.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (8148 ms) 2024-04-04T21:06:29.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-04-04T21:06:29.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (204 ms) 2024-04-04T21:06:29.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (80904 ms total) 2024-04-04T21:06:29.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: 2024-04-04T21:06:29.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [----------] Global test environment tear-down 2024-04-04T21:06:29.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (175955 ms total) 2024-04-04T21:06:29.740 INFO:tasks.workunit.client.0.smithi106.stdout: api_lock: [ PASSED ] 16 tests. 2024-04-04T21:06:31.089 INFO:tasks.workunit.client.0.smithi106.stdout: list: Running main() from gmock_main.cc 2024-04-04T21:06:31.090 INFO:tasks.workunit.client.0.smithi106.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-04-04T21:06:31.090 INFO:tasks.workunit.client.0.smithi106.stdout: list: [----------] Global test environment set-up. 2024-04-04T21:06:31.090 INFO:tasks.workunit.client.0.smithi106.stdout: list: [----------] 3 tests from NeoradosList 2024-04-04T21:06:31.090 INFO:tasks.workunit.client.0.smithi106.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-04-04T21:06:31.090 INFO:tasks.workunit.client.0.smithi106.stdout: list: [ OK ] NeoradosList.ListObjects (39263 ms) 2024-04-04T21:06:31.090 INFO:tasks.workunit.client.0.smithi106.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-04-04T21:06:31.090 INFO:tasks.workunit.client.0.smithi106.stdout: list: [ OK ] NeoradosList.ListObjectsNS (45026 ms) 2024-04-04T21:06:31.090 INFO:tasks.workunit.client.0.smithi106.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-04-04T21:06:31.090 INFO:tasks.workunit.client.0.smithi106.stdout: list: [ OK ] NeoradosList.ListObjectsMany (92637 ms) 2024-04-04T21:06:31.090 INFO:tasks.workunit.client.0.smithi106.stdout: list: [----------] 3 tests from NeoradosList (176926 ms total) 2024-04-04T21:06:31.090 INFO:tasks.workunit.client.0.smithi106.stdout: list: 2024-04-04T21:06:31.091 INFO:tasks.workunit.client.0.smithi106.stdout: list: [----------] Global test environment tear-down 2024-04-04T21:06:31.091 INFO:tasks.workunit.client.0.smithi106.stdout: list: [==========] 3 tests from 1 test suite ran. (176926 ms total) 2024-04-04T21:06:31.091 INFO:tasks.workunit.client.0.smithi106.stdout: list: [ PASSED ] 3 tests. 2024-04-04T21:06:32.694 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: Running main() from gmock_main.cc 2024-04-04T21:06:32.694 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-04-04T21:06:32.694 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [----------] Global test environment set-up. 2024-04-04T21:06:32.694 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-04-04T21:06:32.694 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-04-04T21:06:32.694 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (24483 ms) 2024-04-04T21:06:32.694 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-04-04T21:06:32.694 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-04-04T21:06:32.694 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-04-04T21:06:32.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: no timeout :/ 2024-04-04T21:06:32.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: no timeout :/ 2024-04-04T21:06:32.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: no timeout :/ 2024-04-04T21:06:32.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: no timeout :/ 2024-04-04T21:06:32.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: no timeout :/ 2024-04-04T21:06:32.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (1250 ms) 2024-04-04T21:06:32.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-04-04T21:06:32.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (243 ms) 2024-04-04T21:06:32.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-04-04T21:06:32.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIo.Checksum (80 ms) 2024-04-04T21:06:32.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-04-04T21:06:32.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (141 ms) 2024-04-04T21:06:32.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-04-04T21:06:32.695 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (155 ms) 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (90 ms) 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (28 ms) 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIo.TruncTest (170 ms) 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (78 ms) 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (232 ms) 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIo.RmXattr (2478 ms) 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIo.XattrIter (440 ms) 2024-04-04T21:06:32.696 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [----------] 14 tests from LibRadosIo (29868 ms total) 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (52989 ms) 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (23866 ms) 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (1612 ms) 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (4173 ms) 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (1503 ms) 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (322 ms) 2024-04-04T21:06:32.697 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-04-04T21:06:32.698 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (127 ms) 2024-04-04T21:06:32.698 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-04-04T21:06:32.698 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (745 ms) 2024-04-04T21:06:32.698 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-04-04T21:06:32.698 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (652 ms) 2024-04-04T21:06:32.698 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-04-04T21:06:32.698 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (182 ms) 2024-04-04T21:06:32.698 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [----------] 10 tests from LibRadosIoEC (86171 ms total) 2024-04-04T21:06:32.701 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: 2024-04-04T21:06:32.701 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [----------] Global test environment tear-down 2024-04-04T21:06:32.701 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [==========] 24 tests from 2 test suites ran. (178972 ms total) 2024-04-04T21:06:32.701 INFO:tasks.workunit.client.0.smithi106.stdout: api_io: [ PASSED ] 24 tests. 2024-04-04T21:06:33.805 INFO:tasks.workunit.client.0.smithi106.stdout:25... 2024-04-04T21:06:33.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_15_[72102]: listed object 50... 2024-04-04T21:06:33.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_15_[72102]: listed object 75... 2024-04-04T21:06:33.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_15_[72102]: listed object 100... 2024-04-04T21:06:33.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_15_[72102]: listed object 125... 2024-04-04T21:06:33.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_15_[72102]: saw 150 objects 2024-04-04T21:06:33.805 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_15_[72102]: shutting down. 2024-04-04T21:06:33.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:33.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:33.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:33.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:33.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_16_[72103]: starting. 2024-04-04T21:06:33.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_16_[72103]: added 25 objects... 2024-04-04T21:06:33.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_16_[72103]: added half of the objects 2024-04-04T21:06:33.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_16_[72103]: added 50 objects... 2024-04-04T21:06:33.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_16_[72103]: added 50 objects 2024-04-04T21:06:33.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: process_16_[72103]: shutting down. 2024-04-04T21:06:33.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:33.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:33.806 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:33.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:33.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******************************* 2024-04-04T21:06:33.807 INFO:tasks.workunit.client.0.smithi106.stdout: list_parallel: ******* SUCCESS ********** 2024-04-04T21:06:50.862 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-04-04T21:06:50.862 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-04-04T21:06:50.862 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-04-04T21:06:50.862 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-04-04T21:06:50.863 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ OK ] LibRadosMisc.Exec (26739 ms) 2024-04-04T21:06:50.863 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-04-04T21:06:50.863 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (4626 ms) 2024-04-04T21:06:50.863 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-04-04T21:06:50.863 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (2173 ms) 2024-04-04T21:06:50.863 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-04-04T21:06:50.863 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ OK ] LibRadosMisc.Applications (6373 ms) 2024-04-04T21:06:50.863 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-04-04T21:06:50.863 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-04-04T21:06:50.863 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-04-04T21:06:50.863 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-04-04T21:06:50.863 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-04-04T21:06:50.863 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (70376 ms) 2024-04-04T21:06:50.864 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [----------] 8 tests from LibRadosMisc (110287 ms total) 2024-04-04T21:06:50.864 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: 2024-04-04T21:06:50.864 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [----------] Global test environment tear-down 2024-04-04T21:06:50.864 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (197030 ms total) 2024-04-04T21:06:50.864 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc: [ PASSED ] 12 tests. 2024-04-04T21:07:07.180 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:05:58.697 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:07.184 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:05:58.701 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:07.249 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:58.731 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:07.253 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:58.735 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:07.254 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:05:58.771 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:07.266 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:05:58.782 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:07.395 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:58.878 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:07.404 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:05:58.921 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:07.480 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:05:58.997 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:25.583 INFO:tasks.workunit.client.0.smithi106.stdout: io: Running main() from gmock_main.cc 2024-04-04T21:07:25.584 INFO:tasks.workunit.client.0.smithi106.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-04-04T21:07:25.584 INFO:tasks.workunit.client.0.smithi106.stdout: io: [----------] Global test environment set-up. 2024-04-04T21:07:25.584 INFO:tasks.workunit.client.0.smithi106.stdout: io: [----------] 14 tests from NeoRadosIo 2024-04-04T21:07:25.584 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-04-04T21:07:25.584 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ OK ] NeoRadosIo.Limits (39246 ms) 2024-04-04T21:07:25.584 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-04-04T21:07:25.584 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (28344 ms) 2024-04-04T21:07:25.584 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-04-04T21:07:25.584 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ OK ] NeoRadosIo.ReadOp (17063 ms) 2024-04-04T21:07:25.584 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-04-04T21:07:25.584 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ OK ] NeoRadosIo.SparseRead (22058 ms) 2024-04-04T21:07:25.584 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-04-04T21:07:25.585 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ OK ] NeoRadosIo.RoundTrip (7276 ms) 2024-04-04T21:07:25.585 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-04-04T21:07:25.585 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (8351 ms) 2024-04-04T21:07:25.585 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-04-04T21:07:25.585 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (34056 ms) 2024-04-04T21:07:25.585 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-04-04T21:07:25.585 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (31591 ms) 2024-04-04T21:07:25.585 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-04-04T21:07:25.585 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (13459 ms) 2024-04-04T21:07:25.585 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-04-04T21:07:25.585 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ OK ] NeoRadosIo.Trunc (5278 ms) 2024-04-04T21:07:25.585 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-04-04T21:07:25.585 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ OK ] NeoRadosIo.Remove (7976 ms) 2024-04-04T21:07:25.585 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-04-04T21:07:25.586 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (5640 ms) 2024-04-04T21:07:25.586 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-04-04T21:07:25.586 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ OK ] NeoRadosIo.RmXattr (6250 ms) 2024-04-04T21:07:25.586 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-04-04T21:07:25.586 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ OK ] NeoRadosIo.GetXattrs (4833 ms) 2024-04-04T21:07:25.586 INFO:tasks.workunit.client.0.smithi106.stdout: io: [----------] 14 tests from NeoRadosIo (231422 ms total) 2024-04-04T21:07:25.586 INFO:tasks.workunit.client.0.smithi106.stdout: io: 2024-04-04T21:07:25.586 INFO:tasks.workunit.client.0.smithi106.stdout: io: [----------] Global test environment tear-down 2024-04-04T21:07:25.586 INFO:tasks.workunit.client.0.smithi106.stdout: io: [==========] 14 tests from 1 test suite ran. (231422 ms total) 2024-04-04T21:07:25.586 INFO:tasks.workunit.client.0.smithi106.stdout: io: [ PASSED ] 14 tests. 2024-04-04T21:07:27.530 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:06:19.046 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:27.605 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:06:19.121 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:27.660 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:19.143 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:27.665 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:06:19.182 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:27.665 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:19.148 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:27.669 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:06:19.186 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:27.734 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:19.217 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:27.758 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:06:19.274 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:27.839 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:06:19.356 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:37.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: Running main() from gmock_main.cc 2024-04-04T21:07:37.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-04-04T21:07:37.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [----------] Global test environment set-up. 2024-04-04T21:07:37.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-04-04T21:07:37.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-04-04T21:07:37.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (41997 ms) 2024-04-04T21:07:37.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-04-04T21:07:37.160 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (21074 ms) 2024-04-04T21:07:37.161 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-04-04T21:07:37.161 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (31113 ms) 2024-04-04T21:07:37.161 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-04-04T21:07:37.161 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (8636 ms) 2024-04-04T21:07:37.161 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (102820 ms total) 2024-04-04T21:07:37.161 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: 2024-04-04T21:07:37.161 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-04-04T21:07:37.161 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-04-04T21:07:37.161 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (13509 ms) 2024-04-04T21:07:37.161 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-04-04T21:07:37.161 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (36563 ms) 2024-04-04T21:07:37.161 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-04-04T21:07:37.161 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (38719 ms) 2024-04-04T21:07:37.161 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (88791 ms total) 2024-04-04T21:07:37.162 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: 2024-04-04T21:07:37.162 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-04-04T21:07:37.162 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-04-04T21:07:37.162 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (4168 ms) 2024-04-04T21:07:37.162 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-04-04T21:07:37.162 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (3459 ms) 2024-04-04T21:07:37.162 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-04-04T21:07:37.162 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (3349 ms) 2024-04-04T21:07:37.162 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-04-04T21:07:37.162 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (3497 ms) 2024-04-04T21:07:37.162 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (14473 ms total) 2024-04-04T21:07:37.162 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: 2024-04-04T21:07:37.162 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-04-04T21:07:37.163 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-04-04T21:07:37.163 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (7231 ms) 2024-04-04T21:07:37.163 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-04-04T21:07:37.163 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (7816 ms) 2024-04-04T21:07:37.163 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (15047 ms total) 2024-04-04T21:07:37.163 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: 2024-04-04T21:07:37.163 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [----------] Global test environment tear-down 2024-04-04T21:07:37.163 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (243267 ms total) 2024-04-04T21:07:37.163 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-04-04T21:07:41.621 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: Running main() from gmock_main.cc 2024-04-04T21:07:41.622 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-04-04T21:07:41.622 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [----------] Global test environment set-up. 2024-04-04T21:07:41.622 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-04-04T21:07:41.622 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-04-04T21:07:41.622 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (31959 ms) 2024-04-04T21:07:41.622 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-04-04T21:07:41.622 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (51065 ms) 2024-04-04T21:07:41.622 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-04-04T21:07:41.622 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (19840 ms) 2024-04-04T21:07:41.622 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-04-04T21:07:41.623 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (18078 ms) 2024-04-04T21:07:41.623 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-04-04T21:07:41.623 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (32406 ms) 2024-04-04T21:07:41.623 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-04-04T21:07:41.623 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (34479 ms) 2024-04-04T21:07:41.623 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-04-04T21:07:41.623 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (13512 ms) 2024-04-04T21:07:41.623 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-04-04T21:07:41.623 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (8763 ms) 2024-04-04T21:07:41.623 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-04-04T21:07:41.623 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (4523 ms) 2024-04-04T21:07:41.623 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-04-04T21:07:41.623 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (5694 ms) 2024-04-04T21:07:41.623 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-04-04T21:07:41.624 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (6221 ms) 2024-04-04T21:07:41.624 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-04-04T21:07:41.624 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (10181 ms) 2024-04-04T21:07:41.624 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-04-04T21:07:41.624 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (6070 ms) 2024-04-04T21:07:41.624 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-04-04T21:07:41.624 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (4611 ms) 2024-04-04T21:07:41.624 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (247403 ms total) 2024-04-04T21:07:41.624 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: 2024-04-04T21:07:41.624 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [----------] Global test environment tear-down 2024-04-04T21:07:41.624 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (247403 ms total) 2024-04-04T21:07:41.624 INFO:tasks.workunit.client.0.smithi106.stdout: read_operations: [ PASSED ] 14 tests. 2024-04-04T21:07:47.628 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:06:39.145 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:47.633 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:06:39.150 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:47.684 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:06:39.200 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:47.685 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:06:39.202 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:47.689 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:39.171 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:47.691 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:39.173 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:47.710 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:06:39.227 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:47.750 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:06:39.263 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:47.757 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:39.240 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:07:48.614 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-04-04T21:07:48.614 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-04-04T21:07:48.614 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-04-04T21:07:48.614 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-04-04T21:07:48.614 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: seed 41631 2024-04-04T21:07:48.614 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-04-04T21:07:48.614 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (16000 ms) 2024-04-04T21:07:48.614 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (24421 ms) 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (22592 ms) 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (17911 ms) 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (27277 ms) 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (108201 ms total) 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (13800 ms) 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (59034 ms) 2024-04-04T21:07:48.615 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-04-04T21:07:48.616 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (19882 ms) 2024-04-04T21:07:48.616 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-04-04T21:07:48.616 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (7885 ms) 2024-04-04T21:07:48.616 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-04-04T21:07:48.616 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (4298 ms) 2024-04-04T21:07:48.616 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-04-04T21:07:48.616 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1864-g66deef15/rpm/el9/BUILD/ceph-19.0.0-1864-g66deef15/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-04-04T21:07:48.616 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: 2024-04-04T21:07:48.616 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-04-04T21:07:48.616 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-04-04T21:07:48.616 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi106-41631-7 2024-04-04T21:07:48.616 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: waiting for snaps to purge 2024-04-04T21:07:48.616 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (20191 ms) 2024-04-04T21:07:48.616 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (125090 ms total) 2024-04-04T21:07:48.617 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: 2024-04-04T21:07:48.617 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-04-04T21:07:48.617 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-04-04T21:07:48.617 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (9 ms) 2024-04-04T21:07:48.617 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-04-04T21:07:48.617 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (4885 ms) 2024-04-04T21:07:48.617 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (4894 ms total) 2024-04-04T21:07:48.617 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: 2024-04-04T21:07:48.617 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-04-04T21:07:48.617 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-04-04T21:07:48.617 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3625 ms) 2024-04-04T21:07:48.617 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-04-04T21:08:02.306 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] snapshots: Running main() from gmock_main.cc 2024-04-04T21:08:02.306 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-04-04T21:08:02.306 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [----------] Global test environment set-up. 2024-04-04T21:08:02.306 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-04-04T21:08:02.306 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-04-04T21:08:02.306 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (46165 ms) 2024-04-04T21:08:02.306 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-04-04T21:08:02.306 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (40101 ms) 2024-04-04T21:08:02.306 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-04-04T21:08:02.306 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (26589 ms) 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (10494 ms) 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (69632 ms) 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (192981 ms total) 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (13668 ms) 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (10286 ms) 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (12337 ms) 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (8997 ms) 2024-04-04T21:08:02.307 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-04-04T21:08:02.308 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (9144 ms) 2024-04-04T21:08:02.308 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-04-04T21:08:02.308 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi106-42368-11. 2024-04-04T21:08:02.308 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: Waiting for snaps to purge. 2024-04-04T21:08:02.308 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (20671 ms) 2024-04-04T21:08:02.308 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (75104 ms total) 2024-04-04T21:08:02.308 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: 2024-04-04T21:08:02.308 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [----------] Global test environment tear-down 2024-04-04T21:08:02.308 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (268085 ms total) 2024-04-04T21:08:02.308 INFO:tasks.workunit.client.0.smithi106.stdout: snapshots: [ PASSED ] 11 tests. 2024-04-04T21:08:07.770 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:06:59.287 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:07.775 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:06:59.292 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:07.821 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:59.303 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:07.826 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:59.309 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:07.859 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:06:59.376 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:07.860 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:06:59.377 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:07.862 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:06:59.378 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:07.899 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:59.381 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:07.939 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:06:59.456 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:18.266 INFO:tasks.workunit.client.0.smithi106.stdout: LibRadosSnapshotsECPP.SnapRemovePP (2301 ms) 2024-04-04T21:08:18.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-04-04T21:08:18.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2022 ms) 2024-04-04T21:08:18.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-04-04T21:08:18.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2016 ms) 2024-04-04T21:08:18.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9964 ms total) 2024-04-04T21:08:18.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: 2024-04-04T21:08:18.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-04-04T21:08:18.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-04-04T21:08:18.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (5366 ms) 2024-04-04T21:08:18.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-04-04T21:08:18.267 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (6072 ms) 2024-04-04T21:08:18.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-04-04T21:08:18.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4620 ms) 2024-04-04T21:08:18.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (16058 ms total) 2024-04-04T21:08:18.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: 2024-04-04T21:08:18.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-04-04T21:08:18.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (284367 ms total) 2024-04-04T21:08:18.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-04-04T21:08:18.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-04-04T21:08:18.268 INFO:tasks.workunit.client.0.smithi106.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-04-04T21:08:27.767 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:07:19.284 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:27.839 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:07:19.355 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:27.897 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:19.379 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:27.905 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:07:19.422 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:27.908 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:07:19.424 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:27.910 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:07:19.426 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:27.926 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:19.409 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:27.973 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:07:19.489 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:27.995 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:19.477 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:38.943 INFO:tasks.workunit.client.0.smithi106.stdout: misc: Running main() from gmock_main.cc 2024-04-04T21:08:38.943 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-04-04T21:08:38.943 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [----------] Global test environment set-up. 2024-04-04T21:08:38.943 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-04-04T21:08:38.943 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-04-04T21:08:38.943 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ OK ] NeoRadosMisc.Version (4641 ms) 2024-04-04T21:08:38.943 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-04-04T21:08:38.943 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (4633 ms) 2024-04-04T21:08:38.943 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-04-04T21:08:38.943 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ OK ] NeoRadosMisc.LongName (260978 ms) 2024-04-04T21:08:38.943 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-04-04T21:08:38.944 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (4742 ms) 2024-04-04T21:08:38.944 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-04-04T21:08:38.944 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3355 ms) 2024-04-04T21:08:38.944 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-04-04T21:08:38.944 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (4489 ms) 2024-04-04T21:08:38.944 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-04-04T21:08:38.944 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ OK ] NeoRadosMisc.Exec (3420 ms) 2024-04-04T21:08:38.944 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-04-04T21:08:38.944 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (4537 ms) 2024-04-04T21:08:38.944 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-04-04T21:08:38.944 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (4552 ms) 2024-04-04T21:08:38.944 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-04-04T21:08:38.944 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ OK ] NeoRadosMisc.BigObject (4498 ms) 2024-04-04T21:08:38.945 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-04-04T21:08:38.945 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2162 ms) 2024-04-04T21:08:38.945 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-04-04T21:08:38.945 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (2746 ms) 2024-04-04T21:08:38.945 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [----------] 12 tests from NeoRadosMisc (304754 ms total) 2024-04-04T21:08:38.945 INFO:tasks.workunit.client.0.smithi106.stdout: misc: 2024-04-04T21:08:38.945 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [----------] Global test environment tear-down 2024-04-04T21:08:38.945 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [==========] 12 tests from 1 test suite ran. (304754 ms total) 2024-04-04T21:08:38.945 INFO:tasks.workunit.client.0.smithi106.stdout: misc: [ PASSED ] 12 tests. 2024-04-04T21:08:42.454 INFO:tasks.workunit.client.0.smithi106.stdout:d 2024-04-04T21:08:42.454 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:f905c69b:::2:head expected=8:f905c69b:::2:head 2024-04-04T21:08:42.455 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-04-04T21:08:42.455 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=2 expected=2 2024-04-04T21:08:42.455 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:e960b815:::9:head 2024-04-04T21:08:42.455 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:e960b815:::9:head expected=8:e960b815:::9:head 2024-04-04T21:08:42.455 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-04-04T21:08:42.455 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=9 expected=9 2024-04-04T21:08:42.455 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-04-04T21:08:42.455 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:de5d7c5f:::12:head expected=8:de5d7c5f:::12:head 2024-04-04T21:08:42.455 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-04-04T21:08:42.455 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=12 expected=12 2024-04-04T21:08:42.455 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-04-04T21:08:42.455 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:d83876eb:::4:head expected=8:d83876eb:::4:head 2024-04-04T21:08:42.455 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-04-04T21:08:42.455 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=4 expected=4 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:cfc208b3:::3:head expected=8:cfc208b3:::3:head 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=3 expected=3 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:c4fdafeb:::6:head expected=8:c4fdafeb:::6:head 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=6 expected=6 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:bd63b0f1:::8:head expected=8:bd63b0f1:::8:head 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=8 expected=8 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : cursor()=8:02547ec2:::1:head expected=8:02547ec2:::1:head 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-04-04T21:08:42.456 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: : entry=1 expected=1 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (60227 ms) 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (79904 ms) 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: split 0/5 -> MIN 8:33333333::::head 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: split 1/5 -> 8:33333333::::head 8:66666666::::head 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: split 2/5 -> 8:66666666::::head 8:99999999::::head 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: split 3/5 -> 8:99999999::::head 8:cccccccc::::head 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: split 4/5 -> 8:cccccccc::::head MAX 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (76227 ms) 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [----------] 7 tests from LibRadosList (294081 ms total) 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (2055 ms) 2024-04-04T21:08:42.457 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-04-04T21:08:42.458 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: myset foo1,foo2,foo3 2024-04-04T21:08:42.458 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo1 2024-04-04T21:08:42.458 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo2 2024-04-04T21:08:42.458 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo3 2024-04-04T21:08:42.458 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: myset foo1,foo4,foo5 2024-04-04T21:08:42.458 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo4 2024-04-04T21:08:42.458 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo5 2024-04-04T21:08:42.458 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo1 2024-04-04T21:08:42.458 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: myset foo6,foo7 2024-04-04T21:08:42.458 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo7 2024-04-04T21:08:42.458 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: foo6 2024-04-04T21:08:42.458 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-04-04T21:08:42.458 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: ns1:foo4 2024-04-04T21:08:42.458 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: ns1:foo5 2024-04-04T21:08:42.459 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: ns2:foo7 2024-04-04T21:08:42.459 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: ns2:foo6 2024-04-04T21:08:42.459 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: ns1:foo1 2024-04-04T21:08:42.459 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: :foo1 2024-04-04T21:08:42.459 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: :foo2 2024-04-04T21:08:42.459 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: :foo3 2024-04-04T21:08:42.459 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (4402 ms) 2024-04-04T21:08:42.459 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-04-04T21:08:42.459 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 1 0 2024-04-04T21:08:42.459 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 10 0 2024-04-04T21:08:42.459 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 13 0 2024-04-04T21:08:48.084 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:07:39.601 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:48.085 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:39.567 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:48.089 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:39.571 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:48.091 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:07:39.608 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:48.140 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:07:39.657 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:48.153 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:07:39.669 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:48.157 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:39.639 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:48.157 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:07:39.674 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:48.236 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:07:39.753 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:08:50.408 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 7 0 2024-04-04T21:08:50.408 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 14 0 2024-04-04T21:08:50.408 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 0 0 2024-04-04T21:08:50.408 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 15 0 2024-04-04T21:08:50.408 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 11 0 2024-04-04T21:08:50.408 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 5 0 2024-04-04T21:08:50.408 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 8 0 2024-04-04T21:08:50.408 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 6 0 2024-04-04T21:08:50.408 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 3 0 2024-04-04T21:08:50.408 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 4 0 2024-04-04T21:08:50.408 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 12 0 2024-04-04T21:08:50.408 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 9 0 2024-04-04T21:08:50.409 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 2 0 2024-04-04T21:08:50.409 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-04-04T21:08:50.409 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (2889 ms) 2024-04-04T21:08:50.409 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [----------] 3 tests from LibRadosListEC (9346 ms total) 2024-04-04T21:08:50.409 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 2024-04-04T21:08:50.409 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-04-04T21:08:50.409 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-04-04T21:08:50.409 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3384 ms) 2024-04-04T21:08:50.409 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [----------] 1 test from LibRadosListNP (3384 ms total) 2024-04-04T21:08:50.409 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: 2024-04-04T21:08:50.409 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [----------] Global test environment tear-down 2024-04-04T21:08:50.409 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [==========] 11 tests from 3 test suites ran. (316649 ms total) 2024-04-04T21:08:50.409 INFO:tasks.workunit.client.0.smithi106.stdout: api_list: [ PASSED ] 11 tests. 2024-04-04T21:08:56.825 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: Running main() from gmock_main.cc 2024-04-04T21:08:56.825 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-04-04T21:08:56.825 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [----------] Global test environment set-up. 2024-04-04T21:08:56.825 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-04-04T21:08:56.825 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-04-04T21:08:56.825 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (40437 ms) 2024-04-04T21:08:56.825 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-04-04T21:08:56.826 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (22320 ms) 2024-04-04T21:08:56.826 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-04-04T21:08:56.826 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (20264 ms) 2024-04-04T21:08:56.826 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-04-04T21:08:56.826 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (35741 ms) 2024-04-04T21:08:56.826 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-04-04T21:08:56.826 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (5640 ms) 2024-04-04T21:08:56.826 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-04-04T21:08:56.826 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (28931 ms) 2024-04-04T21:08:56.826 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-04-04T21:08:56.826 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (169252 ms) 2024-04-04T21:08:56.826 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (322585 ms total) 2024-04-04T21:08:56.827 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: 2024-04-04T21:08:56.827 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [----------] Global test environment tear-down 2024-04-04T21:08:56.827 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (322585 ms total) 2024-04-04T21:08:56.827 INFO:tasks.workunit.client.0.smithi106.stdout: write_operations: [ PASSED ] 7 tests. 2024-04-04T21:09:07.953 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-04-04T21:09:07.954 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-04-04T21:09:07.954 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-04-04T21:09:07.954 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-04-04T21:09:07.954 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-04-04T21:09:07.954 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-04-04T21:09:07.954 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: 2024-04-04T21:09:07.954 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-04-04T21:09:07.954 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: seed 41351 2024-04-04T21:09:07.954 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-04-04T21:09:07.954 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (191 ms) 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (24289 ms) 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (1216 ms) 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (272 ms) 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (339 ms) 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (34 ms) 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (35 ms) 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (237 ms) 2024-04-04T21:09:07.955 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-04-04T21:09:07.956 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (76 ms) 2024-04-04T21:09:07.956 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-04-04T21:09:07.956 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (2644 ms) 2024-04-04T21:09:07.956 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-04-04T21:09:07.956 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (578 ms) 2024-04-04T21:09:07.956 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-04-04T21:09:07.956 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (1165 ms) 2024-04-04T21:09:07.956 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-04-04T21:09:07.956 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (207 ms) 2024-04-04T21:09:07.956 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-04-04T21:09:07.956 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: osd_max_attr_size = 0 2024-04-04T21:09:07.956 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-04-04T21:09:07.956 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (198914 ms) 2024-04-04T21:09:07.956 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-04-04T21:09:07.957 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (14439 ms) 2024-04-04T21:09:07.957 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-04-04T21:09:07.957 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-04-04T21:09:07.957 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: done waiting, doing copies 2024-04-04T21:09:07.957 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: waiting for final deep scrubs... 2024-04-04T21:09:07.957 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: done waiting 2024-04-04T21:09:07.957 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (78895 ms) 2024-04-04T21:09:07.957 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-04-04T21:09:07.957 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (315 ms) 2024-04-04T21:09:07.957 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-04-04T21:09:07.957 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (55 ms) 2024-04-04T21:09:07.957 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-04-04T21:09:07.957 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5012 ms) 2024-04-04T21:09:07.957 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-04-04T21:09:07.958 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-04-04T21:09:07.958 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-04-04T21:09:07.958 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-04-04T21:09:07.958 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-04-04T21:09:07.958 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-04-04T21:09:08.223 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:07:59.740 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:08.225 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:07:59.742 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:08.266 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:07:59.783 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:08.276 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:07:59.792 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:08.277 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:07:59.794 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:08.313 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:59.795 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:08.315 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:59.798 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:08.325 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:07:59.842 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:08.375 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:59.857 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:13.698 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] api_c_write_operations: Running main() from gmock_main.cc 2024-04-04T21:09:13.698 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-04-04T21:09:13.698 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-04-04T21:09:13.698 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-04-04T21:09:13.698 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-04-04T21:09:13.699 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-04-04T21:09:13.699 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-04-04T21:09:13.699 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: 2024-04-04T21:09:13.699 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-04-04T21:09:13.699 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-04-04T21:09:13.699 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (40755 ms) 2024-04-04T21:09:13.699 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-04-04T21:09:13.699 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (38330 ms) 2024-04-04T21:09:13.699 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-04-04T21:09:13.699 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (27746 ms) 2024-04-04T21:09:13.699 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-04-04T21:09:13.699 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (58055 ms) 2024-04-04T21:09:13.699 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-04-04T21:09:13.700 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (28216 ms) 2024-04-04T21:09:13.700 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-04-04T21:09:13.700 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (8456 ms) 2024-04-04T21:09:13.700 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-04-04T21:09:13.700 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (138116 ms) 2024-04-04T21:09:13.700 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (339674 ms total) 2024-04-04T21:09:13.700 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: 2024-04-04T21:09:13.700 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-04-04T21:09:13.700 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (339675 ms total) 2024-04-04T21:09:13.700 INFO:tasks.workunit.client.0.smithi106.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-04-04T21:09:28.290 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:08:19.806 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:28.351 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:08:19.868 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:28.355 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:19.838 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:28.386 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:19.868 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:28.411 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:08:19.928 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:28.437 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:19.920 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:28.477 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:08:19.993 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:28.482 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:08:19.999 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:28.543 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:08:20.059 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:39.627 INFO:tasks.workunit.client.0.smithi106.stdout: 22 tests from LibRadosMiscPP (328913 ms total) 2024-04-04T21:09:39.627 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: 2024-04-04T21:09:39.627 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-04-04T21:09:39.627 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-04-04T21:09:39.627 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (2067 ms) 2024-04-04T21:09:39.627 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (2067 ms total) 2024-04-04T21:09:39.627 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: 2024-04-04T21:09:39.627 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-04-04T21:09:39.628 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-04-04T21:09:39.628 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (504 ms) 2024-04-04T21:09:39.628 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-04-04T21:09:39.628 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (112 ms) 2024-04-04T21:09:39.628 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (616 ms total) 2024-04-04T21:09:39.628 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: 2024-04-04T21:09:39.628 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-04-04T21:09:39.628 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-04-04T21:09:39.628 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (531 ms) 2024-04-04T21:09:39.628 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-04-04T21:09:39.628 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (49 ms) 2024-04-04T21:09:39.628 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (580 ms total) 2024-04-04T21:09:39.628 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: 2024-04-04T21:09:39.629 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-04-04T21:09:39.629 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-04-04T21:09:39.629 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (599 ms) 2024-04-04T21:09:39.629 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-04-04T21:09:39.629 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (44 ms) 2024-04-04T21:09:39.629 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (643 ms total) 2024-04-04T21:09:39.629 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: 2024-04-04T21:09:39.629 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-04-04T21:09:39.629 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-04-04T21:09:39.629 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1648 ms) 2024-04-04T21:09:39.629 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1648 ms total) 2024-04-04T21:09:39.629 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: 2024-04-04T21:09:39.630 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-04-04T21:09:39.630 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (365827 ms total) 2024-04-04T21:09:39.630 INFO:tasks.workunit.client.0.smithi106.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-04-04T21:09:48.235 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:08:39.751 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:48.237 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:08:39.754 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:48.269 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:08:39.786 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:48.285 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:39.767 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:48.286 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:39.769 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:48.287 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:08:39.803 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:48.288 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:08:39.804 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:48.321 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:39.804 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:48.324 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:08:39.840 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:52.989 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:08:44.506 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:52.993 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:08:44.510 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:53.008 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:08:44.525 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:53.041 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:44.523 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:53.043 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:44.525 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:53.046 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:08:44.562 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:53.047 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:08:44.564 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:53.058 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:44.541 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:09:53.063 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:08:44.580 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:08.492 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:59.974 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:08.494 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:59.977 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:08.494 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:09:00.011 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:08.497 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:09:00.014 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:08.526 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:00.008 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:08.532 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:09:00.049 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:08.575 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:09:00.091 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:08.579 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:09:00.095 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:08.618 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:09:00.134 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:28.432 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:09:19.949 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:28.435 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:09:19.952 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:28.478 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:09:19.995 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:28.481 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:19.963 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:28.483 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:19.966 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:28.486 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:09:20.003 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:28.489 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:09:20.006 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:28.525 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:20.008 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:28.530 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:09:20.046 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:48.502 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:09:40.018 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:48.506 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:09:40.023 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:48.547 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:09:40.064 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:48.553 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:40.036 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:48.555 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:40.037 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:48.557 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:09:40.074 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:48.559 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:09:40.075 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:48.593 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:40.076 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:10:48.600 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:09:40.117 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:08.799 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:10:00.316 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:08.801 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:10:00.318 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:08.802 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:00.285 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:08.806 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:00.288 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:08.844 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:10:00.361 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:08.847 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:00.329 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:08.852 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:10:00.369 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:08.856 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:10:00.373 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:08.895 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:10:00.412 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:28.890 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:10:20.406 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:28.944 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:10:20.461 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:28.987 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:10:20.504 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:28.994 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:20.476 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:28.996 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:20.478 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:28.997 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:10:20.514 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:28.999 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:10:20.516 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:29.036 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:20.519 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:29.058 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:10:20.575 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:48.820 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:10:40.336 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:48.823 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:10:40.340 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:48.848 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:10:40.365 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:48.870 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:10:40.386 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:48.875 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:10:40.391 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:48.878 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:40.361 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:48.882 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:40.365 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:48.903 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:10:40.417 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:11:48.911 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:40.393 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:09.060 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:11:00.577 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:09.064 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:11:00.581 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:09.104 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:11:00.621 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:09.117 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:00.600 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:09.119 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:11:00.636 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:09.120 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:00.602 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:09.124 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:11:00.641 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:09.155 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:00.638 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:09.159 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:11:00.676 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:10.274 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:11:01.791 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:10.326 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:11:01.843 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:10.340 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:11:01.858 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:10.378 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:01.860 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:10.382 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:01.864 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:10.390 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:11:01.907 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:10.393 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:11:01.909 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:10.394 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:01.877 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:10.410 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:11:01.927 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:29.028 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:11:20.544 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:29.080 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:11:20.597 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:29.120 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:11:20.637 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:29.130 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:20.612 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:29.134 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:20.616 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:29.134 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:11:20.650 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:29.136 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:11:20.653 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:29.179 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:20.661 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:29.181 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:11:20.698 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:49.153 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:11:40.668 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:49.154 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:11:40.671 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:49.158 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:40.641 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:49.167 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:40.650 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:49.203 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:11:40.720 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:49.211 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:40.694 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:49.215 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:11:40.731 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:49.217 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:11:40.734 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:12:49.262 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:11:40.778 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:04.202 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-04-04T21:13:04.202 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-04-04T21:13:04.202 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-04-04T21:13:04.202 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: seed 41404 2024-04-04T21:13:04.203 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-04-04T21:13:04.203 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (21713 ms) 2024-04-04T21:13:04.203 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-04-04T21:13:04.203 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (63049 ms) 2024-04-04T21:13:04.203 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-04-04T21:13:04.203 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (632 ms) 2024-04-04T21:13:04.203 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (85394 ms total) 2024-04-04T21:13:04.203 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: 2024-04-04T21:13:04.203 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-04-04T21:13:04.203 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-04-04T21:13:04.203 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (100206 ms) 2024-04-04T21:13:04.203 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-04-04T21:13:04.203 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (33569 ms) 2024-04-04T21:13:04.203 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-04-04T21:13:04.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (36049 ms) 2024-04-04T21:13:04.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-04-04T21:13:04.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: my_snaps [3] 2024-04-04T21:13:04.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: my_snaps [4,3] 2024-04-04T21:13:04.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: my_snaps [5,4,3] 2024-04-04T21:13:04.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-04-04T21:13:04.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: promoting some heads 2024-04-04T21:13:04.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: promoting from clones for snap 6 2024-04-04T21:13:04.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: promoting from clones for snap 5 2024-04-04T21:13:04.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: promoting from clones for snap 4 2024-04-04T21:13:04.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: promoting from clones for snap 3 2024-04-04T21:13:04.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: waiting for scrubs... 2024-04-04T21:13:04.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: done waiting 2024-04-04T21:13:04.204 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (130839 ms) 2024-04-04T21:13:04.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-04-04T21:13:04.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (12180 ms) 2024-04-04T21:13:04.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-04-04T21:13:04.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (9048 ms) 2024-04-04T21:13:04.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-04-04T21:13:04.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (10119 ms) 2024-04-04T21:13:04.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-04-04T21:13:04.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (10554 ms) 2024-04-04T21:13:04.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-04-04T21:13:04.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (13125 ms) 2024-04-04T21:13:04.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-04-04T21:13:04.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (10365 ms) 2024-04-04T21:13:04.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-04-04T21:13:04.205 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (12245 ms) 2024-04-04T21:13:04.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-04-04T21:13:04.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (37871 ms) 2024-04-04T21:13:04.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-04-04T21:13:04.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (10535 ms) 2024-04-04T21:13:04.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-04-04T21:13:04.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (12389 ms) 2024-04-04T21:13:04.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-04-04T21:13:04.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (15072 ms) 2024-04-04T21:13:04.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-04-04T21:13:04.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (9777 ms) 2024-04-04T21:13:04.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-04-04T21:13:04.206 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (13268 ms) 2024-04-04T21:13:09.227 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:12:00.744 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:09.231 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:12:00.748 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:09.273 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:12:00.790 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:09.280 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:12:00.797 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:09.281 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:12:00.798 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:09.293 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:00.775 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:09.294 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:00.777 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:09.327 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:12:00.844 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:09.332 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:00.814 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:25.460 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-04-04T21:13:25.460 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: hmm, no HitSet yet 2024-04-04T21:13:25.460 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: ok, hit_set contains 175:602f83fe:::foo:head 2024-04-04T21:13:25.460 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9335 ms) 2024-04-04T21:13:25.461 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-04-04T21:13:25.461 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg_num = 32 2024-04-04T21:13:25.461 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 0 ls 1712265200,0 2024-04-04T21:13:25.461 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 1 ls 1712265200,0 2024-04-04T21:13:25.461 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 2 ls 1712265200,0 2024-04-04T21:13:25.461 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 3 ls 1712265200,0 2024-04-04T21:13:25.461 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 4 ls 1712265200,0 2024-04-04T21:13:25.461 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 5 ls 1712265200,0 2024-04-04T21:13:25.461 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 6 ls 1712265200,0 2024-04-04T21:13:25.461 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 7 ls 1712265200,0 2024-04-04T21:13:25.461 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 8 ls 1712265200,0 2024-04-04T21:13:25.461 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 9 ls 1712265200,0 2024-04-04T21:13:25.461 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 10 ls 1712265200,0 2024-04-04T21:13:25.461 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 11 ls 1712265200,0 2024-04-04T21:13:25.462 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 12 ls 1712265200,0 2024-04-04T21:13:25.462 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 13 ls 1712265200,0 2024-04-04T21:13:25.462 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 14 ls 1712265200,0 2024-04-04T21:13:25.462 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 15 ls 1712265200,0 2024-04-04T21:13:25.462 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 16 ls 1712265200,0 2024-04-04T21:13:25.462 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 17 ls 1712265200,0 2024-04-04T21:13:25.462 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 18 ls 1712265200,0 2024-04-04T21:13:25.462 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 19 ls 1712265200,0 2024-04-04T21:13:25.462 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 20 ls 1712265200,0 2024-04-04T21:13:25.462 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 21 ls 1712265200,0 2024-04-04T21:13:25.462 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 22 ls 1712265200,0 2024-04-04T21:13:25.462 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 23 ls 1712265200,0 2024-04-04T21:13:25.462 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 24 ls 1712265200,0 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 25 ls 1712265200,0 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 26 ls 1712265200,0 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 27 ls 1712265200,0 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 28 ls 1712265200,0 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 29 ls 1712265200,0 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 30 ls 1712265200,0 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg 31 ls 1712265200,0 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: pg_num = 32 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:6cac518f:::0:head 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:02547ec2:::1:head 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:f905c69b:::2:head 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:cfc208b3:::3:head 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:d83876eb:::4:head 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:b29083e3:::5:head 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:c4fdafeb:::6:head 2024-04-04T21:13:25.463 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:5c6b0b28:::7:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:bd63b0f1:::8:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:e960b815:::9:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:52ea6a34:::10:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:89d3ae78:::11:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:de5d7c5f:::12:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:566253c9:::13:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:62a1935d:::14:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:863748b0:::15:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:3958e169:::16:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:4d4dabf9:::17:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:8391935d:::18:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:28883081:::19:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:69259c59:::20:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:4bdb80b7:::21:head 2024-04-04T21:13:25.464 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:a11c5d71:::22:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:271af37b:::23:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:95b121be:::24:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:58d1031b:::25:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:0a050783:::26:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:c709704c:::27:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:cbe56eaf:::28:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:86b4b162:::29:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:70d89383:::30:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:dd450c7c:::31:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:6d5729b1:::32:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:c388f3fb:::33:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:56cfea31:::34:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:9dbc1bf7:::35:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:40b74ccd:::36:head 2024-04-04T21:13:25.465 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:4d5aaf42:::37:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:920f362c:::38:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:6cc53222:::39:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:9cad833f:::40:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:1ea84d41:::41:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:c4480ef6:::42:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:a694361e:::43:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:d1bd33e9:::44:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:ddc2cd5d:::45:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:2b782207:::46:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:7b187fca:::47:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:90ecdf6f:::48:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:a5ed95fe:::49:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:ea0eaa55:::50:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:f33ef17b:::51:head 2024-04-04T21:13:25.466 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:a0d1b2f6:::52:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:60c5229e:::53:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:edcbc575:::54:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:102cf253:::55:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:efb7fb0b:::56:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:50d0a326:::57:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:d4dc5daf:::58:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:3a130462:::59:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:ec87ed71:::60:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:d5bc9454:::61:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:3ddfe313:::62:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:7c2816b9:::63:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:47e00e4d:::64:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:c6410c18:::65:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:b48ed237:::66:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:cd63ad31:::67:head 2024-04-04T21:13:25.467 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:b179e92b:::68:head 2024-04-04T21:13:25.468 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:0d9f741a:::69:head 2024-04-04T21:13:25.468 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:6d3352ae:::70:head 2024-04-04T21:13:25.468 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:c6d5c19e:::71:head 2024-04-04T21:13:25.468 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:bc4729c3:::72:head 2024-04-04T21:13:25.468 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:77e930b9:::73:head 2024-04-04T21:13:25.468 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:0abeecfd:::74:head 2024-04-04T21:13:25.468 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:b7c37e15:::75:head 2024-04-04T21:13:25.468 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:b6378398:::76:head 2024-04-04T21:13:25.468 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:02bd68de:::77:head 2024-04-04T21:13:25.468 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:cc795d2d:::78:head 2024-04-04T21:13:25.468 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:630d4fea:::79:head 2024-04-04T21:13:25.468 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:e0d29ef5:::80:head 2024-04-04T21:13:25.468 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:fd6f13d2:::81:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:606461d5:::82:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:eadbdc43:::83:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:8761d0bb:::84:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:9ef0186f:::85:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:e0d41294:::86:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:961de695:::87:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:1423148f:::88:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:633a8fa2:::89:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:a8653809:::90:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:3dac8b33:::91:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:35aad435:::92:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:f6dcc343:::93:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:dbbdad87:::94:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:1cb48ce0:::95:head 2024-04-04T21:13:25.469 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:03cd461c:::96:head 2024-04-04T21:13:25.470 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:17a4ea99:::97:head 2024-04-04T21:13:25.470 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:9993c9a7:::98:head 2024-04-04T21:13:25.470 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:6394211c:::99:head 2024-04-04T21:13:25.470 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:94c7ae57:::100:head 2024-04-04T21:13:25.470 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:6fdee5bb:::101:head 2024-04-04T21:13:25.470 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:9a477fd1:::102:head 2024-04-04T21:13:25.470 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:eb850916:::103:head 2024-04-04T21:13:25.470 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:affc56b9:::104:head 2024-04-04T21:13:25.470 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:b42dc814:::105:head 2024-04-04T21:13:25.470 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:f319f8f0:::106:head 2024-04-04T21:13:25.470 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:9a40b9de:::107:head 2024-04-04T21:13:25.470 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:8b524f28:::108:head 2024-04-04T21:13:25.470 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:e3de589f:::109:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:90f90a5b:::110:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:a7b4f1d7:::111:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:af51766e:::112:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:b6f90bd1:::113:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:e0261208:::114:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:c9569ef7:::115:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:61bebe50:::116:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:fe93412b:::117:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:d3d38bee:::118:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:3100ba0c:::119:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:d0560ada:::120:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:f0ea8b35:::121:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:766f231a:::122:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:a07a2582:::123:head 2024-04-04T21:13:25.471 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:bd7c6b3a:::124:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:fb2ddaff:::125:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:4408e1fe:::126:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:ee1df7a7:::127:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:c3002909:::128:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:4f48ffa9:::129:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:edf38733:::130:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:c08425c0:::131:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:5f902d98:::132:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:41ea2c93:::133:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:813cee13:::134:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:0131818d:::135:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:26ba5a85:::136:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:381b8a5a:::137:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:28797e47:::138:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:bfca7f22:::139:head 2024-04-04T21:13:25.472 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:36807075:::140:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:80b03975:::141:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:5c15709b:::142:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:f39ea15e:::143:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:ea992956:::144:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:48887b1c:::145:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:9f24a9dd:::146:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:987f100b:::147:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:d2dd3581:::148:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:7fed1808:::149:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:c80b70e9:::150:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:85ed90f9:::151:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:36428b24:::152:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:d044c34a:::153:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:7c18bf58:::154:head 2024-04-04T21:13:25.473 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:d1c21232:::155:head 2024-04-04T21:13:25.474 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:a7a3c575:::156:head 2024-04-04T21:13:25.474 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:87da0633:::157:head 2024-04-04T21:13:25.474 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:d5ac3822:::158:head 2024-04-04T21:13:25.474 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:3f20522d:::159:head 2024-04-04T21:13:25.474 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:6ca26563:::160:head 2024-04-04T21:13:25.474 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:532ce135:::161:head 2024-04-04T21:13:25.474 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:c78863e6:::162:head 2024-04-04T21:13:29.259 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:12:20.776 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:29.322 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:12:20.839 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:29.358 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:12:20.876 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:29.371 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:20.853 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:29.377 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:20.859 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:29.380 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:12:20.896 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:29.381 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:12:20.898 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:29.412 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:20.895 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:29.418 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:12:20.935 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:49.270 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:12:40.787 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:49.273 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:12:40.791 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:49.303 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:12:40.820 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:49.321 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:40.804 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:49.322 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:12:40.838 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:49.323 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:12:40.840 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:49.324 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:40.807 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:49.352 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:12:40.869 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:13:49.353 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:40.835 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:04.396 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:12:55.913 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:04.464 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:12:55.981 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:04.496 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:12:56.013 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:04.517 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:55.999 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:04.518 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:56.000 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:04.524 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:12:56.041 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:04.526 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:12:56.043 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:04.546 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:56.029 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:04.568 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:12:56.084 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:14.524 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:06.041 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:14.532 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:06.049 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:14.566 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:06.083 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:14.586 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:06.103 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:14.592 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:06.108 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:14.600 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:06.082 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:14.606 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:06.088 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:14.632 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:06.148 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:14.644 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:06.126 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:19.387 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:10.904 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:19.389 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:10.906 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:19.404 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:10.921 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:19.436 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:10.952 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:19.438 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:10.954 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:19.448 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:10.930 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:19.449 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:10.932 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:19.454 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:10.970 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:19.470 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:10.953 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:29.183 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:20.699 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:29.187 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:20.705 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:29.206 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:20.723 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:29.238 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:20.720 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:29.239 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:20.721 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:29.255 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:20.771 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:29.256 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:20.773 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:29.257 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:20.740 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:29.276 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:20.790 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:34.439 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:25.956 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:34.440 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:25.922 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:34.442 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:25.959 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:34.442 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:25.925 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:34.456 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:25.938 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:34.459 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:25.977 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:34.493 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:26.010 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:34.496 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:26.012 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:34.511 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:26.028 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:35.118 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:26.600 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:35.119 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:26.602 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:35.128 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:26.610 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:35.183 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:26.699 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:35.184 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:26.701 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:35.193 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:26.709 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:35.218 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:26.735 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:35.219 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:26.736 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:35.227 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:26.744 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:35.886 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:7462ddf6:::163:head 2024-04-04T21:14:35.886 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:5d165639:::164:head 2024-04-04T21:14:35.886 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:f43765fc:::165:head 2024-04-04T21:14:35.886 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:b4c720e9:::166:head 2024-04-04T21:14:35.886 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:e694b040:::167:head 2024-04-04T21:14:35.887 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:afa38db2:::168:head 2024-04-04T21:14:35.887 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:77ba9f53:::169:head 2024-04-04T21:14:35.887 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:87495034:::170:head 2024-04-04T21:14:35.887 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:7c96bf0e:::171:head 2024-04-04T21:14:35.887 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:dbe346cc:::172:head 2024-04-04T21:14:35.887 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:e943ec24:::173:head 2024-04-04T21:14:35.887 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:f97a9c0c:::174:head 2024-04-04T21:14:35.887 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:6f26e74d:::175:head 2024-04-04T21:14:35.887 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:4f95e106:::176:head 2024-04-04T21:14:35.887 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:0e6f2f8f:::177:head 2024-04-04T21:14:35.887 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:05db05f1:::178:head 2024-04-04T21:14:35.887 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:38a78d66:::179:head 2024-04-04T21:14:35.887 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:d095610b:::180:head 2024-04-04T21:14:35.888 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:a1a9d709:::181:head 2024-04-04T21:14:35.888 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:1e5d39db:::182:head 2024-04-04T21:14:35.888 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:f7df4fb9:::183:head 2024-04-04T21:14:35.888 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:03a7f161:::184:head 2024-04-04T21:14:35.888 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:ba70721e:::185:head 2024-04-04T21:14:35.888 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:28e5662d:::186:head 2024-04-04T21:14:35.888 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:973d52de:::187:head 2024-04-04T21:14:35.888 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:4303eb1c:::188:head 2024-04-04T21:14:35.888 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:b990b48e:::189:head 2024-04-04T21:14:35.888 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:29b8165b:::190:head 2024-04-04T21:14:35.888 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:3547f197:::191:head 2024-04-04T21:14:35.888 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:7e260936:::192:head 2024-04-04T21:14:35.888 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:1abec7b1:::193:head 2024-04-04T21:14:35.888 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:10fdda93:::194:head 2024-04-04T21:14:35.889 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:15817eea:::195:head 2024-04-04T21:14:35.889 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:770bab57:::196:head 2024-04-04T21:14:35.889 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:ed9e13e7:::197:head 2024-04-04T21:14:35.889 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:71471a8f:::198:head 2024-04-04T21:14:35.889 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: checking for 176:10fb1d02:::199:head 2024-04-04T21:14:35.889 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (20727 ms) 2024-04-04T21:14:35.889 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-04-04T21:14:35.889 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265222,0 2024-04-04T21:14:35.889 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: first is 1712265222 2024-04-04T21:14:35.889 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265222,0 2024-04-04T21:14:35.889 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265222,0 2024-04-04T21:14:35.889 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265222,1712265224,1712265225,0 2024-04-04T21:14:35.889 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265222,1712265224,1712265225,0 2024-04-04T21:14:35.889 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265222,1712265224,1712265225,0 2024-04-04T21:14:35.890 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265222,1712265224,1712265225,1712265228,1712265229,0 2024-04-04T21:14:35.890 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265222,1712265224,1712265225,1712265228,1712265229,0 2024-04-04T21:14:35.890 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265222,1712265224,1712265225,1712265228,1712265229,0 2024-04-04T21:14:35.890 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265222,1712265224,1712265225,1712265228,1712265229,1712265231,1712265232,0 2024-04-04T21:14:35.890 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265222,1712265224,1712265225,1712265228,1712265229,1712265231,1712265232,0 2024-04-04T21:14:35.890 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265222,1712265224,1712265225,1712265228,1712265229,1712265231,1712265232,0 2024-04-04T21:14:35.890 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265225,1712265228,1712265229,1712265231,1712265232,1712265234,1712265235,0 2024-04-04T21:14:35.890 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: first now 1712265225, trimmed 2024-04-04T21:14:35.890 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21893 ms) 2024-04-04T21:14:35.890 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-04-04T21:14:35.890 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: foo0 2024-04-04T21:14:35.890 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-04-04T21:14:35.891 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (18112 ms) 2024-04-04T21:14:35.891 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-04-04T21:14:49.599 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:41.116 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:49.660 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:41.122 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:49.660 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:41.163 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:49.661 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:41.172 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:49.661 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:41.177 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:49.668 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:41.150 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:49.672 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:41.154 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:49.694 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:41.210 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:49.710 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:41.193 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:54.504 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:46.021 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:54.506 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:46.024 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:54.517 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:46.035 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:54.553 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:46.070 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:54.554 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:46.037 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:54.556 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:46.073 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:54.556 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:46.038 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:54.570 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:46.052 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:14:54.570 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:46.087 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:04.502 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:56.019 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:04.555 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:56.072 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:04.578 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:13:56.095 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:04.605 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:56.122 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:04.609 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:56.126 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:04.623 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:56.105 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:04.626 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:56.108 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:04.633 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:13:56.150 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:04.650 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:13:56.132 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:14.590 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:06.107 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:14.592 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:06.109 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:14.618 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:06.136 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:14.643 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:06.160 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:14.644 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:06.161 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:14.648 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:06.131 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:14.650 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:06.133 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:14.671 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:06.187 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:14.682 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:06.165 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:19.639 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:11.156 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:19.688 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:11.205 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:19.712 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:11.229 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:19.739 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:11.221 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:19.740 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:11.223 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:19.756 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:11.238 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:19.756 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:11.272 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:19.768 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:11.285 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:19.788 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:11.305 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:34.658 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:26.175 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:34.663 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:26.181 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:34.704 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:26.221 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:34.727 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:26.209 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:34.730 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:26.213 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:34.738 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:26.254 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:34.739 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:26.256 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:34.767 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:26.249 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:34.779 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:26.296 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:35.480 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:26.962 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:35.481 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:26.963 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:35.488 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:26.971 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:35.524 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:27.041 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:35.525 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:27.042 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:35.533 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:27.050 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:35.552 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:27.069 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:35.553 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:27.070 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:35.561 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:27.079 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:54.877 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:46.394 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:54.881 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:46.398 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:54.917 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:14:46.434 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:54.932 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:46.415 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:54.935 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:46.418 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:54.942 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:46.459 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:54.945 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:46.462 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:54.957 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:14:46.439 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:15:54.968 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:14:46.485 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:14.850 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:06.366 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:14.851 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:06.368 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:14.883 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:06.400 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:14.897 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:06.380 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:14.898 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:06.381 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:14.901 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:06.418 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:14.903 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:06.420 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:14.929 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:06.411 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:14.936 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:06.453 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:25.072 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:16.589 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:25.077 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:16.594 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:25.098 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:16.615 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:25.131 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:16.613 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:25.134 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:16.651 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:25.136 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:16.619 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:25.140 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:16.657 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:25.157 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:16.674 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:25.157 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:16.640 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:34.864 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:26.381 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:34.868 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:26.386 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:34.892 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:26.410 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:34.916 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:26.399 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:34.917 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:26.434 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:34.918 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:26.400 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:34.918 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:26.435 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:34.942 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:26.424 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:34.947 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:26.464 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:40.060 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:31.577 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:40.076 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:31.594 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:40.104 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:31.621 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:40.122 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:31.638 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:40.130 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:31.612 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:40.131 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:31.614 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:40.131 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:31.648 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:40.151 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:31.634 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:40.155 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:31.672 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:54.926 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:46.442 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:54.972 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:46.489 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:54.989 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:46.507 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:55.020 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:46.503 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:55.025 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:46.508 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:55.026 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:46.542 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:55.027 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:46.544 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:55.043 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:46.526 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:55.045 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:46.561 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:59.969 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:51.486 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:59.973 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:51.490 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:16:59.988 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:15:51.505 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:00.021 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:51.503 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:00.024 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:51.506 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:00.038 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:51.521 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:00.040 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:51.557 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:00.057 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:51.574 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:00.070 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:15:51.586 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.027 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:06.544 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.028 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:06.546 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.058 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:06.575 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.073 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:06.556 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.073 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:06.590 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.078 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:06.595 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.078 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:06.561 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.111 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:06.594 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.113 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:06.629 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.741 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:07.223 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.741 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:07.224 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.749 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:07.232 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.762 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:07.279 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.763 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:07.280 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.772 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:07.289 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.794 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:07.310 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.795 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:07.312 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:15.803 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:07.319 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:30.114 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:21.632 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:30.132 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:21.649 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:30.163 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:21.646 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:30.164 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:21.681 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:30.165 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:21.647 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:30.166 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:21.682 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:30.168 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:21.684 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:30.195 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:21.678 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:30.199 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:21.716 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:40.152 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:31.669 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:40.156 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:31.673 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:40.181 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:31.698 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:40.204 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:31.686 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:40.206 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:31.689 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:40.207 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:31.724 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:40.213 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:31.729 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:40.228 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:31.711 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:40.235 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:31.752 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:45.168 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:36.685 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:45.176 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:36.693 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:45.187 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:36.705 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:45.220 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:36.737 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:45.224 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:36.706 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:45.226 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:36.743 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:45.228 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:36.710 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:45.239 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:36.722 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:45.241 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:36.758 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:17:58.126 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:17:58.112+0000 26283640 -1 snap_mapper.add_oid found existing snaps mapped on 194:e4b3421e:test-rados-api-smithi106-41404-80::foo:2, removing 2024-04-04T21:17:58.141 INFO:tasks.ceph.osd.0.smithi106.stderr:2024-04-04T21:17:58.134+0000 27e8b640 -1 snap_mapper.add_oid found existing snaps mapped on 194:e4b3421e:test-rados-api-smithi106-41404-80::foo:2, removing 2024-04-04T21:18:00.374 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:51.891 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:00.376 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:51.893 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:00.397 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:51.914 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:00.432 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:51.949 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:00.434 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:51.951 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:00.447 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:51.929 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:00.453 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:51.935 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:00.460 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:51.977 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:00.479 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:51.962 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:05.263 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:56.780 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:05.267 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:56.784 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:05.282 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:16:56.799 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:05.313 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:56.795 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:05.314 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:56.796 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:05.322 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:56.839 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:05.324 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:56.841 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:05.328 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:16:56.810 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:05.343 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:16:56.860 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:20.528 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:12.044 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:20.534 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:12.052 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:20.569 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:12.087 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:20.582 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:12.064 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:20.583 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:12.066 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:20.601 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:12.118 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:20.603 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:12.120 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:20.616 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:12.099 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:20.657 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:12.174 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:31.187 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (21618 ms) 2024-04-04T21:18:31.188 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-04-04T21:18:31.188 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (25226 ms) 2024-04-04T21:18:31.188 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-04-04T21:18:31.188 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4478 ms) 2024-04-04T21:18:31.188 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-04-04T21:18:31.188 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.188 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (5440 ms) 2024-04-04T21:18:31.188 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-04-04T21:18:31.188 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (5604 ms) 2024-04-04T21:18:31.188 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-04-04T21:18:31.188 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.189 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4620 ms) 2024-04-04T21:18:31.189 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-04-04T21:18:31.189 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.189 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5435 ms) 2024-04-04T21:18:31.189 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-04-04T21:18:31.189 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.189 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40330 ms) 2024-04-04T21:18:31.189 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-04-04T21:18:31.189 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.189 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17862 ms) 2024-04-04T21:18:31.189 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-04-04T21:18:31.189 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.189 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5798 ms) 2024-04-04T21:18:31.190 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-04-04T21:18:31.190 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.190 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4648 ms) 2024-04-04T21:18:31.190 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-04-04T21:18:31.190 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.190 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (26978 ms) 2024-04-04T21:18:31.190 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-04-04T21:18:31.190 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.190 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (17139 ms) 2024-04-04T21:18:31.190 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-04-04T21:18:31.190 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.190 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6992 ms) 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5726 ms) 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: waiting for scrubs... 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: done waiting 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26954 ms) 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (12055 ms) 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9238 ms) 2024-04-04T21:18:31.191 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-04-04T21:18:31.192 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:18:31.192 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (10778 ms) 2024-04-04T21:18:35.612 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:27.129 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:35.618 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:27.136 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:35.680 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:27.197 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:35.682 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:27.165 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:35.686 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:27.168 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:35.687 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:27.204 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:35.692 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:27.209 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:35.735 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:27.252 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:35.737 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:27.220 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:40.603 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:32.120 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:40.607 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:32.124 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:40.623 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:32.141 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:40.661 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:32.177 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:40.661 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:32.144 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:40.662 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:32.179 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:40.664 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:32.147 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:40.680 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:32.197 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:40.681 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:32.163 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:50.454 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:41.971 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:50.536 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:42.053 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:50.571 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:42.088 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:50.599 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:42.116 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:50.600 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:42.083 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:50.609 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:42.126 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:50.610 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:42.093 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:50.645 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:42.162 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:18:50.651 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:42.133 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:00.711 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:52.227 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:00.715 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:52.233 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:00.740 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:52.257 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:00.764 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:52.280 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:00.764 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:52.247 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:00.768 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:52.250 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:00.769 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:52.286 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:00.792 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:52.275 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:00.793 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:52.310 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:05.579 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:57.096 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:05.583 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:57.100 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:05.597 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:17:57.114 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:05.629 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:57.112 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:05.630 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:57.146 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:05.631 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:57.148 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:05.631 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:57.113 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:05.641 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:17:57.124 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:05.643 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:17:57.160 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:20.683 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:12.200 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:20.685 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:12.203 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:20.695 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:12.212 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:20.704 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:12.221 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:20.738 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:12.221 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:20.749 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:12.232 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:20.777 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:12.260 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:20.786 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:12.303 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:20.797 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:12.314 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:25.676 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:17.194 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:25.679 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:17.196 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:25.696 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:17.214 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:25.720 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:17.203 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:25.721 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:17.204 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:25.725 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:17.242 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:25.727 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:17.244 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:25.736 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:17.218 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:25.744 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:17.261 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:30.007 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:21.524 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:30.012 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:21.530 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:30.027 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:21.544 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:30.064 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:21.547 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:30.066 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:21.582 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:30.066 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:21.549 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:30.080 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:21.563 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:30.084 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:21.601 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:30.109 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:21.626 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:36.267 INFO:tasks.ceph.osd.1.smithi106.stderr:2024-04-04T21:19:36.260+0000 29a8a640 -1 snap_mapper.add_oid found existing snaps mapped on 97:4f66d269:test-rados-api-smithi106-41404-97::foo:21, removing 2024-04-04T21:19:36.270 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-04-04T21:19:36.242+0000 2227b640 -1 snap_mapper.add_oid found existing snaps mapped on 97:4f66d269:test-rados-api-smithi106-41404-97::foo:21, removing 2024-04-04T21:19:40.711 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:32.228 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:40.718 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:32.235 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:40.746 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:32.263 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:40.764 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:32.246 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:40.767 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:32.250 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:40.780 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:32.297 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:40.785 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:32.302 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:40.786 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:32.269 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:40.816 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:32.333 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:41.436 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:32.919 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:41.437 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:32.920 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:41.445 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:32.928 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:41.626 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:33.142 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:41.688 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:33.144 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:41.688 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:33.154 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:41.730 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:33.247 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:41.740 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:33.258 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:41.750 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:33.267 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:55.772 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:47.289 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:55.774 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:47.292 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:55.810 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:47.327 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:55.821 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:47.303 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:55.822 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:47.305 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:55.824 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:47.341 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:55.825 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:47.342 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:55.856 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:47.338 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:19:55.860 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:47.377 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:00.795 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:52.312 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:00.797 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:52.315 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:00.826 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:18:52.343 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:00.841 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:52.324 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:00.847 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:52.329 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:00.863 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:18:52.345 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:00.869 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:52.386 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:00.875 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:52.392 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:00.891 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:18:52.408 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:11.168 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:02.685 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:11.172 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:02.690 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:11.215 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:02.733 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:11.236 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:02.719 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:11.238 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:02.720 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:11.246 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:02.763 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:11.247 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:02.764 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:11.279 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:02.762 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:11.289 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:02.806 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:11.974 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:03.457 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:11.976 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:03.459 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:11.991 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:03.474 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:12.011 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:03.529 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:12.013 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:03.531 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:12.031 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:03.549 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:12.215 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:03.731 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:12.216 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:03.733 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:12.231 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:03.747 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:21.303 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:12.820 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:21.312 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:12.829 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:21.353 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:12.870 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:21.382 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:12.864 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:21.383 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:12.866 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:21.384 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:12.901 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:21.390 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:12.906 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:21.435 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:12.917 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:21.443 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:12.960 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:26.200 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:17.717 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:26.205 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:17.722 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:26.231 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:17.749 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:26.247 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:17.730 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:26.250 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:17.733 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:26.267 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:17.783 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:26.268 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:17.750 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:26.271 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:17.788 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:26.288 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:17.805 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:41.208 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:32.725 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:41.260 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:32.778 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:41.297 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:32.814 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:41.308 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:32.825 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:41.310 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:32.827 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:41.313 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:32.795 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:41.316 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:32.799 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:41.355 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:32.872 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:41.358 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:32.839 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:42.196 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:33.678 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:42.197 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:33.680 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:42.208 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:33.690 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:42.281 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:33.798 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:42.283 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:33.800 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:42.298 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:33.814 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:42.314 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:33.831 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:42.315 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:33.832 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:42.326 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:33.844 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:56.331 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:47.848 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:56.381 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:47.897 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:56.382 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:47.865 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:56.382 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:47.899 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:56.387 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:47.870 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:56.388 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:47.905 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:56.423 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:47.939 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:56.426 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:47.908 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:20:56.427 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:47.944 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:03.624 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlu ec_list: Running main() from gmock_main.cc 2024-04-04T21:21:03.754 INFO:tasks.workunit.client.0.smithi106.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-04-04T21:21:03.754 INFO:tasks.workunit.client.0.smithi106.stdout: ec_list: [----------] Global test environment set-up. 2024-04-04T21:21:03.755 INFO:tasks.workunit.client.0.smithi106.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-04-04T21:21:03.755 INFO:tasks.workunit.client.0.smithi106.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-04-04T21:21:03.755 INFO:tasks.workunit.client.0.smithi106.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (993818 ms) 2024-04-04T21:21:03.755 INFO:tasks.workunit.client.0.smithi106.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-04-04T21:21:03.755 INFO:tasks.workunit.client.0.smithi106.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (9364 ms) 2024-04-04T21:21:03.755 INFO:tasks.workunit.client.0.smithi106.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-04-04T21:21:03.755 INFO:tasks.workunit.client.0.smithi106.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (46262 ms) 2024-04-04T21:21:03.755 INFO:tasks.workunit.client.0.smithi106.stdout: ec_list: [----------] 3 tests from NeoradosECList (1049444 ms total) 2024-04-04T21:21:03.755 INFO:tasks.workunit.client.0.smithi106.stdout: ec_list: 2024-04-04T21:21:03.755 INFO:tasks.workunit.client.0.smithi106.stdout: ec_list: [----------] Global test environment tear-down 2024-04-04T21:21:03.755 INFO:tasks.workunit.client.0.smithi106.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (1049444 ms total) 2024-04-04T21:21:03.755 INFO:tasks.workunit.client.0.smithi106.stdout: ec_list: [ PASSED ] 3 tests. 2024-04-04T21:21:06.586 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:58.103 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:06.593 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:58.110 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:06.634 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:19:58.152 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:06.662 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:58.179 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:06.664 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:58.181 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:06.665 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:58.148 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:06.668 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:58.151 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:06.709 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:19:58.191 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:06.710 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:19:58.227 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:11.550 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:03.067 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:11.553 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:03.070 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:11.568 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:03.085 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:11.601 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:03.084 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:11.602 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:03.085 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:11.614 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:03.130 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:11.615 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:03.132 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:11.616 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:03.099 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:11.629 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:03.146 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:26.497 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:18.015 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:26.502 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:18.019 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:26.537 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:18.052 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:26.550 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:18.067 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:26.553 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:18.070 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:26.554 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:18.036 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:26.555 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:18.038 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:26.582 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:18.099 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:26.583 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:18.066 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:27.187 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:18.670 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:27.188 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:18.671 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:27.196 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:18.678 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:27.216 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:18.733 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:27.217 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:18.734 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:27.218 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:18.736 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:27.220 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:18.737 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:27.225 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:18.742 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:27.228 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:18.745 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:39.337 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:30.854 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:39.344 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:30.862 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:39.355 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:30.838 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:39.358 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:30.841 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:39.370 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:30.887 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:39.381 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:30.898 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:39.437 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:30.919 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:39.447 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:30.965 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:39.494 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:31.011 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:46.702 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:38.219 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:46.705 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:38.222 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:46.739 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:38.256 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:46.753 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:38.235 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:46.755 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:38.237 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:46.776 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:38.259 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:46.782 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:38.298 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:46.784 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:38.301 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:21:46.806 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:38.323 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:01.652 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:53.169 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:01.656 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:53.174 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:01.690 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:53.207 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:01.712 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:53.195 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:01.714 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:53.196 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:01.738 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:53.255 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:01.739 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:53.256 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:01.746 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:53.229 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:01.775 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:53.292 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:06.668 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:58.185 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:06.670 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:58.187 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:06.692 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:20:58.209 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:06.717 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:58.199 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:06.718 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:58.235 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:06.720 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:58.202 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:06.721 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:58.238 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:06.736 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:20:58.219 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:06.738 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:20:58.255 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:16.706 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:08.223 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:16.707 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:08.225 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:16.731 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:08.248 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:16.751 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:08.234 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:16.752 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:08.235 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:16.756 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:08.273 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:16.757 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:08.274 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:16.775 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:08.258 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:16.781 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:08.298 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:26.877 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:18.394 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:26.880 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:18.397 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:26.918 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:18.435 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:26.952 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:18.469 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:26.954 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:18.437 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:26.955 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:18.472 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:26.956 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:18.439 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:26.990 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:18.472 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:26.999 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:18.516 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:31.856 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:23.373 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:31.859 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:23.377 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:31.876 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:23.393 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:31.905 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:23.388 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:31.906 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:23.389 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:31.917 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:23.434 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:31.918 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:23.435 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:31.921 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:23.404 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:31.934 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:23.451 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:34.674 INFO:tasks.workunit.client.0.smithi106.stdout:sh 2024-04-04T21:22:34.674 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:22:34.674 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9299 ms) 2024-04-04T21:22:34.674 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-04-04T21:22:34.674 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:22:34.674 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (7560 ms) 2024-04-04T21:22:34.674 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-04-04T21:22:34.675 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:22:34.675 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6851 ms) 2024-04-04T21:22:34.675 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-04-04T21:22:34.675 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:22:34.675 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26957 ms) 2024-04-04T21:22:34.675 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-04-04T21:22:34.675 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:22:34.675 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15274 ms) 2024-04-04T21:22:34.675 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-04-04T21:22:34.675 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12850 ms) 2024-04-04T21:22:34.675 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-04-04T21:22:34.675 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-04-04T21:22:34.675 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: 00000010 69 73 |is| 2024-04-04T21:22:34.676 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: 00000012 2024-04-04T21:22:34.676 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (13209 ms) 2024-04-04T21:22:34.676 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-04-04T21:22:34.676 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: require_osd_release = squid 2024-04-04T21:22:34.676 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7050 ms) 2024-04-04T21:22:34.676 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (903247 ms total) 2024-04-04T21:22:34.676 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: 2024-04-04T21:22:34.676 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-04-04T21:22:34.676 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-04-04T21:22:34.676 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1940 ms) 2024-04-04T21:22:34.676 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-04-04T21:22:34.676 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (12229 ms) 2024-04-04T21:22:34.676 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-04-04T21:22:34.676 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (20017 ms) 2024-04-04T21:22:34.677 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-04-04T21:22:34.677 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (80 ms) 2024-04-04T21:22:34.677 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (34266 ms total) 2024-04-04T21:22:34.677 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: 2024-04-04T21:22:34.677 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-04-04T21:22:34.677 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-04-04T21:22:34.677 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (9385 ms) 2024-04-04T21:22:34.677 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-04-04T21:22:34.677 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (9025 ms) 2024-04-04T21:22:34.677 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-04-04T21:22:34.677 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: waiting for scrub... 2024-04-04T21:22:34.677 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: done waiting 2024-04-04T21:22:34.677 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (26315 ms) 2024-04-04T21:22:34.678 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-04-04T21:22:34.678 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (11209 ms) 2024-04-04T21:22:34.678 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-04-04T21:22:34.678 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (9376 ms) 2024-04-04T21:22:34.678 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-04-04T21:22:34.678 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (10690 ms) 2024-04-04T21:22:34.678 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-04-04T21:22:34.678 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (13171 ms) 2024-04-04T21:22:34.678 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-04-04T21:22:46.870 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:38.387 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:46.919 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:38.401 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:46.921 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:38.404 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:46.921 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:38.439 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:46.949 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:38.431 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:46.952 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:38.469 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:46.969 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:38.486 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:46.973 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:38.490 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:47.014 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:38.531 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:47.680 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:39.163 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:47.681 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:39.164 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:47.692 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:39.174 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:47.862 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:39.380 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:47.864 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:39.381 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:47.865 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:39.382 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:47.866 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:39.383 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:47.873 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:39.390 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:22:47.875 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:39.392 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:02.053 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:53.570 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:02.057 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:53.574 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:02.109 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:53.626 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:02.124 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:53.607 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:02.125 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:53.642 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:02.126 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:53.609 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:02.145 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:53.662 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:02.171 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:53.654 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:02.203 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:53.719 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:07.051 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:58.568 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:07.054 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:58.572 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:07.069 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:21:58.587 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:07.099 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:58.581 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:07.100 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:58.582 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:07.101 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:58.618 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:07.102 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:58.619 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:07.115 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:21:58.597 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:07.117 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:21:58.634 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:17.080 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:08.597 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:17.082 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:08.600 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:17.116 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:08.634 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:17.127 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:08.610 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:17.131 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:08.614 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:17.133 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:08.650 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:17.136 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:08.653 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:17.154 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:08.636 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:17.160 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:08.676 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:27.126 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:18.644 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:27.130 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:18.648 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:27.155 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:18.673 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:27.176 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:18.659 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:27.177 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:18.660 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:27.179 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:18.696 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:27.181 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:18.698 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:27.204 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:18.687 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:27.206 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:18.723 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:32.177 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:23.694 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:32.192 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:23.709 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:32.220 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:23.703 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:32.225 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:23.743 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:32.226 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:23.709 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:32.227 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:23.744 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:32.231 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:23.748 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:32.244 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:23.727 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:32.249 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:23.766 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:45.573 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:37.090 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:45.578 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:37.096 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:45.592 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:37.074 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:45.617 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:37.134 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:45.640 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:37.123 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:45.650 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:37.167 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:45.651 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:37.168 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:45.674 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:37.157 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:45.694 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:37.211 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:47.355 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:38.872 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:47.357 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:38.875 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:47.370 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:38.888 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:47.406 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:38.889 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:47.407 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:38.924 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:47.408 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:38.925 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:47.408 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:38.891 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:47.417 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:38.900 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:23:47.420 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:38.937 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:02.260 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:53.776 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:02.262 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:53.779 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:02.289 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:53.807 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:02.308 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:53.791 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:02.309 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:53.792 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:02.310 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:53.827 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:02.312 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:53.829 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:02.337 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:53.820 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:02.353 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:53.870 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:07.230 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:58.747 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:07.276 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:58.794 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:07.281 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:58.764 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:07.283 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:58.765 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:07.288 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:22:58.805 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:07.294 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:22:58.777 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:07.323 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:58.840 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:07.324 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:58.841 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:07.335 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:22:58.851 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:17.403 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:08.921 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:17.454 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:08.971 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:17.491 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:09.008 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:17.516 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:08.998 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:17.520 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:09.002 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:17.539 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:09.056 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:17.541 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:09.058 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:17.551 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:09.034 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:17.568 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:09.085 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:32.469 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:23.985 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:32.480 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:23.962 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:32.482 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:23.999 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:32.530 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:24.047 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:32.549 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:24.032 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:32.550 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:24.067 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:32.565 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:24.082 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:32.597 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:24.079 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:32.600 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:24.117 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:37.593 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:29.110 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:37.595 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:29.112 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:37.614 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:29.131 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:37.641 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:29.124 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:37.642 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:29.125 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:37.647 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:29.164 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:37.649 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:29.166 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:37.655 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:29.138 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:37.662 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:29.179 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:52.583 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:44.100 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:52.586 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:44.103 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:52.620 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:44.137 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:52.640 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:44.157 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:52.645 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:44.162 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:52.645 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:44.128 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:52.649 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:44.132 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:52.670 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:44.187 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:24:52.674 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:44.157 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:07.536 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:59.053 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:07.537 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:59.054 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:07.567 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:23:59.084 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:07.586 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:59.069 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:07.587 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:59.070 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:07.598 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:59.115 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:07.600 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:59.117 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:07.613 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:23:59.096 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:07.626 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:23:59.143 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:12.554 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:04.071 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:12.557 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:04.074 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:12.570 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:04.087 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:12.602 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:04.085 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:12.603 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:04.086 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:12.612 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:04.128 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:12.613 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:04.130 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:12.618 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:04.100 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:12.625 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:04.142 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:22.729 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:14.247 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:22.735 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:14.252 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:22.769 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:14.287 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:22.799 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:14.315 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:22.801 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:14.284 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:22.801 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:14.318 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:22.803 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:14.286 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:22.834 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:14.351 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:22.837 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:14.320 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:32.783 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:24.300 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:32.787 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:24.304 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:32.828 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:24.345 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:32.849 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:24.332 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:32.856 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:24.373 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:32.859 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:24.342 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:32.861 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:24.378 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:32.897 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:24.413 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:32.897 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:24.380 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:37.777 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:29.294 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:37.784 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:29.301 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:37.807 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:29.324 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:37.837 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:29.320 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:37.841 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:29.323 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:37.843 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:29.360 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:37.847 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:29.364 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:37.862 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:29.345 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:37.869 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:29.386 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:38.341 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (9794 ms) 2024-04-04T21:25:38.341 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-04-04T21:25:38.341 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14976 ms) 2024-04-04T21:25:38.341 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-04-04T21:25:38.341 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (9944 ms) 2024-04-04T21:25:38.341 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-04-04T21:25:38.341 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (17951 ms) 2024-04-04T21:25:38.341 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-04-04T21:25:38.342 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (10307 ms) 2024-04-04T21:25:38.342 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-04-04T21:25:38.342 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (10977 ms) 2024-04-04T21:25:38.342 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-04-04T21:25:38.342 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: hmm, no HitSet yet 2024-04-04T21:25:38.342 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: ok, hit_set contains 228:602f83fe:::foo:head 2024-04-04T21:25:38.342 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9911 ms) 2024-04-04T21:25:38.342 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-04-04T21:25:38.342 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265837,0 2024-04-04T21:25:38.342 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: first is 1712265837 2024-04-04T21:25:38.342 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265837,0 2024-04-04T21:25:38.342 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265837,0 2024-04-04T21:25:38.342 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265837,1712265840,1712265841,0 2024-04-04T21:25:38.342 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265837,1712265840,1712265841,0 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265837,1712265840,1712265841,0 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265837,1712265840,1712265841,1712265843,1712265844,0 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265837,1712265840,1712265841,1712265843,1712265844,0 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265837,1712265840,1712265841,1712265843,1712265844,0 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265837,1712265840,1712265841,1712265843,1712265844,1712265846,1712265847,0 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265837,1712265840,1712265841,1712265843,1712265844,1712265846,1712265847,0 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265837,1712265840,1712265841,1712265843,1712265844,1712265846,1712265847,0 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: got ls 1712265841,1712265843,1712265844,1712265846,1712265847,1712265849,1712265850,0 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: first now 1712265841, trimmed 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22618 ms) 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: foo0 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (16524 ms) 2024-04-04T21:25:38.343 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (21497 ms) 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (25226 ms) 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (6395 ms) 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3967 ms) 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (5475 ms) 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4629 ms) 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (279362 ms total) 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-04-04T21:25:38.344 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1324501 ms total) 2024-04-04T21:25:38.345 INFO:tasks.workunit.client.0.smithi106.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-04-04T21:25:38.346 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:25:38.346 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41165 2024-04-04T21:25:38.346 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41165 2024-04-04T21:25:38.346 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:25:38.346 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41357 2024-04-04T21:25:38.346 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41357 2024-04-04T21:25:38.346 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:25:38.346 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41863 2024-04-04T21:25:38.346 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41863 2024-04-04T21:25:38.346 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:25:38.347 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41602 2024-04-04T21:25:38.347 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41602 2024-04-04T21:25:38.347 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:25:38.347 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41425 2024-04-04T21:25:38.347 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41425 2024-04-04T21:25:38.347 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:25:38.347 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41263 2024-04-04T21:25:38.347 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41263 2024-04-04T21:25:38.347 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:25:38.347 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41534 2024-04-04T21:25:38.347 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41534 2024-04-04T21:25:38.682 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:30.165 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:38.683 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:30.166 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:38.692 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:30.175 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:38.757 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:30.274 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:38.757 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:30.274 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:38.758 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:30.275 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:38.758 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:30.275 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:38.768 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:30.285 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:38.768 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:30.285 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:52.775 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:44.292 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:52.824 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:44.341 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:52.847 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:44.364 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:52.878 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:44.359 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:52.878 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:44.360 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:52.881 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:44.398 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:52.885 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:44.402 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:52.894 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:44.377 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:52.902 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:44.419 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:57.821 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:49.338 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:57.823 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:49.341 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:57.834 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:24:49.351 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:57.870 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:49.353 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:57.871 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:49.388 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:57.872 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:49.355 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:57.872 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:49.389 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:57.882 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:24:49.399 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:25:57.882 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:24:49.365 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:12.809 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:04.326 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:12.856 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:04.373 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:12.860 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:04.343 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:12.861 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:04.344 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:12.881 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:04.364 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:12.882 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:04.399 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:12.913 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:04.430 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:12.917 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:04.434 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:12.935 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:04.452 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:16.078 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:07.595 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:16.082 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:07.600 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:16.097 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:07.614 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:16.133 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:07.616 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:16.134 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:07.617 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:16.146 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:07.662 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:16.148 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:07.631 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:16.165 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:07.682 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:16.179 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:07.696 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:17.869 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:09.386 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:17.872 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:09.389 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:17.881 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:09.398 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:17.917 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:09.400 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:17.919 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:09.435 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:17.919 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:09.402 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:17.920 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:09.437 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:17.927 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:09.410 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:17.929 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:09.446 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:32.902 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:24.419 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:32.904 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:24.422 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:32.919 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:24.437 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:32.953 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:24.436 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:32.954 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:24.470 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:32.954 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:24.437 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:32.957 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:24.473 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:32.970 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:24.453 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:32.972 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:24.489 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:37.914 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:29.431 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:37.917 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:29.434 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:37.932 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:29.449 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:37.965 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:29.448 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:37.966 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:29.449 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:37.968 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:29.485 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:37.969 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:29.486 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:37.978 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:29.460 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:37.979 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:29.496 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:52.970 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:44.486 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:52.972 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:44.489 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:52.986 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:44.504 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:53.017 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:44.499 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:53.019 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:44.502 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:53.020 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:44.537 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:53.022 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:44.539 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:53.035 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:44.518 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:53.037 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:44.554 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:57.940 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:49.457 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:57.990 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:49.507 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:58.003 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:25:49.521 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:58.036 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:49.553 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:58.041 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:49.524 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:58.041 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:49.558 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:58.043 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:49.526 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:58.054 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:25:49.571 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:26:58.055 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:25:49.538 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:13.017 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:04.534 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:13.020 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:04.537 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:13.038 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:04.555 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:13.067 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:04.550 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:13.067 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:04.584 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:13.071 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:04.554 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:13.071 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:04.588 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:13.088 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:04.571 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:13.089 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:04.606 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:18.029 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:09.546 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:18.033 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:09.551 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:18.044 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:09.561 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:18.079 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:09.595 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:18.081 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:09.598 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:18.082 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:09.564 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:18.083 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:09.566 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:18.092 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:09.609 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:18.095 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:09.577 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:23.016 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:14.533 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:23.062 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:14.579 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:23.080 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:14.597 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:23.113 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:14.630 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:23.114 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:14.597 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:23.115 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:14.632 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:23.119 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:14.602 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:23.129 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:14.645 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:23.131 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:14.614 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.048 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:29.565 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.095 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:29.613 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.115 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:29.632 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.140 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:29.623 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.141 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:29.624 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.147 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:29.664 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.149 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:29.666 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.158 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:29.640 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.187 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:29.704 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.753 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:30.235 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.753 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:30.236 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.761 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:30.244 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.800 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:30.318 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.801 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:30.319 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.809 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:30.327 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.824 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:30.341 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.825 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:30.342 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:38.836 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:30.353 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.329 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:49.845 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.334 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:49.851 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.354 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:49.871 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.379 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:49.861 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.380 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:49.863 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.384 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:49.901 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.385 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:49.902 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.398 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:49.880 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.405 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:49.921 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.936 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:50.419 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.937 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:50.420 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.945 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:50.428 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.949 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:50.467 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.951 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:50.468 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.958 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:50.476 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.981 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:50.498 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.983 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:50.499 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:27:58.990 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:50.507 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:03.317 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:54.833 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:03.320 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:54.838 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:03.332 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:26:54.850 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:03.367 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:54.884 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:03.368 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:54.885 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:03.371 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:54.854 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:03.372 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:54.855 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:03.380 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:26:54.897 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:03.383 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:26:54.866 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.346 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:09.864 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.349 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:09.867 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.371 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:09.889 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.398 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:09.881 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.402 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:09.885 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.403 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:09.920 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.404 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:09.921 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.419 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:09.902 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.421 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:09.938 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.972 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:10.455 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.973 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:10.456 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.982 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:10.465 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.989 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:10.506 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.990 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:10.508 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.993 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:10.510 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.994 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:10.511 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:18.999 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:10.516 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:19.002 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:10.519 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:23.320 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:14.837 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:23.367 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:14.884 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:23.385 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:14.902 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:23.413 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:14.896 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:23.415 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:14.898 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:23.426 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:14.909 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:23.431 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:14.948 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:23.434 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:14.951 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:23.445 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:14.962 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:38.398 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:29.915 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:38.402 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:29.919 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:38.419 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:29.936 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:38.449 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:29.932 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:38.451 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:29.934 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:38.463 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:29.980 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:38.465 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:29.982 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:38.468 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:29.951 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:38.482 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:29.999 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:39.010 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:30.493 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:39.011 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:30.494 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:39.018 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:30.501 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:39.031 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:30.549 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:39.033 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:30.550 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:39.041 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:30.558 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:39.055 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:30.572 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:39.057 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:30.574 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:39.065 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:30.582 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:43.418 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:34.935 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:43.420 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:34.938 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:43.433 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:34.950 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:43.467 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:34.950 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:43.468 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:34.984 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:43.468 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:34.951 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:43.469 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:34.986 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:43.479 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:34.962 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:43.480 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:34.997 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:58.457 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:49.974 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:58.459 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:49.977 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:58.474 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:49.991 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:58.507 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:50.023 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:58.508 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:50.025 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:58.509 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:49.992 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:58.510 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:49.993 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:58.524 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:50.041 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:58.575 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:50.057 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:59.071 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:50.554 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:59.072 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:50.555 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:59.080 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:27:50.563 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:59.109 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:50.627 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:59.110 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:50.628 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:59.118 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:27:50.636 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:59.132 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:50.649 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:59.133 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:50.650 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:28:59.141 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:27:50.658 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:13.497 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:05.014 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:13.499 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:05.016 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:13.523 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:05.040 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:13.546 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:05.029 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:13.548 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:05.031 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:13.556 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:05.073 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:13.557 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:05.074 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:13.565 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:05.047 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:13.574 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:05.091 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:18.501 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:10.018 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:18.506 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:10.024 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:18.518 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:10.035 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:18.550 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:10.033 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:18.551 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:10.068 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:18.553 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:10.036 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:18.554 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:10.071 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:18.564 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:10.047 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:18.564 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:10.081 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:19.087 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:10.569 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:19.088 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:10.571 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:19.096 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:10.579 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:19.113 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:10.631 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:19.114 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:10.632 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:19.121 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:10.638 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:19.122 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:10.639 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:19.122 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:10.640 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:19.130 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:10.647 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:38.516 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:30.033 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:38.563 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:30.080 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:38.584 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:30.101 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:38.609 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:30.126 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:38.612 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:30.129 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:38.617 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:30.100 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:38.619 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:30.102 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:38.632 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:30.148 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:38.637 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:30.120 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:39.191 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:30.673 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:39.192 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:30.674 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:39.193 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:30.711 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:39.194 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:30.712 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:39.199 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:30.682 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:39.205 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:30.723 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:39.207 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:30.724 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:39.208 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:30.725 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:39.216 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:30.733 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:58.555 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:50.072 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:58.621 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:50.139 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:58.643 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:50.160 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:58.667 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:50.150 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:58.674 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:50.155 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:58.675 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:50.192 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:58.677 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:50.194 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:58.694 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:50.176 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:58.703 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:50.219 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:59.279 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:50.762 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:59.281 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:50.764 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:59.291 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:50.808 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:59.292 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:50.810 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:59.293 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:28:50.776 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:59.305 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:50.822 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:59.306 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:50.823 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:59.307 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:28:50.824 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:29:59.319 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:28:50.836 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:18.647 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:10.164 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:18.650 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:10.168 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:18.669 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:10.187 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:18.696 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:10.179 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:18.697 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:10.180 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:18.698 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:10.215 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:18.702 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:10.219 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:18.716 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:10.199 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:18.720 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:10.237 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:19.177 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:10.660 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:19.178 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:10.661 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:19.186 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:10.669 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:19.199 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:10.716 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:19.200 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:10.717 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:19.208 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:10.725 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:19.208 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:10.725 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:19.209 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:10.726 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:19.217 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:10.734 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:38.640 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:30.158 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:38.689 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:30.206 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:38.692 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:30.175 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:38.696 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:30.179 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:38.712 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:30.229 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:38.717 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:30.200 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:38.740 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:30.257 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:38.741 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:30.258 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:38.760 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:30.277 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:39.189 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:30.672 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:39.190 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:30.673 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:39.197 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:30.680 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:39.275 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:30.792 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:39.276 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:30.794 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:39.282 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:30.799 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:39.284 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:30.801 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:39.290 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:30.808 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:39.293 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:30.810 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:58.680 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:50.197 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:58.729 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:50.246 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:58.730 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:50.248 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:58.731 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:50.213 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:58.731 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:50.247 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:58.734 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:50.217 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:58.754 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:50.271 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:58.755 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:50.237 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:58.758 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:50.275 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:59.234 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:50.716 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:59.235 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:50.717 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:59.242 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:29:50.725 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:59.273 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:50.790 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:59.274 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:50.791 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:59.282 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:29:50.798 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:59.322 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:50.839 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:59.323 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:50.840 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:30:59.330 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:29:50.848 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:18.729 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:10.245 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:18.773 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:10.290 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:18.796 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:10.313 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:18.823 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:10.306 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:18.824 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:10.307 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:18.826 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:10.342 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:18.828 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:10.346 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:18.844 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:10.327 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:18.848 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:10.365 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:19.228 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:10.745 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:19.229 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:10.747 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:19.247 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:10.729 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:19.247 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:10.730 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:19.255 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:10.738 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:19.273 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:10.790 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:19.273 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:10.791 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:19.274 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:10.791 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:19.282 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:10.799 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:38.817 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:30.334 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:38.818 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:30.335 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:38.836 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:30.353 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:38.871 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:30.388 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:38.872 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:30.389 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:38.873 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:30.355 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:38.874 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:30.357 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:38.891 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:30.408 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:38.892 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:30.375 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:39.438 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:30.920 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:39.440 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:30.922 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:39.447 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:30.929 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:39.467 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:30.984 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:39.468 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:30.985 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:39.468 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:30.985 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:39.469 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:30.987 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:39.480 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:30.996 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:39.480 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:30.997 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:58.859 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:50.376 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:58.860 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:50.378 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:58.878 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:50.396 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:58.909 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:50.426 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:58.910 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:50.427 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:58.913 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:50.396 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:58.915 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:50.398 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:58.929 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:50.446 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:58.933 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:50.416 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:59.436 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:50.919 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:59.437 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:50.920 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:59.445 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:30:50.928 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:59.461 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:50.978 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:59.462 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:50.980 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:59.463 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:50.980 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:59.464 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:50.981 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:59.472 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:30:50.990 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:31:59.474 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:30:50.991 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:18.904 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:10.422 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:18.906 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:10.424 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:18.924 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:10.441 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:18.954 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:10.437 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:18.955 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:10.472 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:18.956 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:10.473 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:18.957 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:10.440 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:18.978 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:10.461 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:18.981 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:10.498 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:19.473 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:10.955 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:19.473 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:10.956 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:19.481 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:10.964 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:19.513 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:11.031 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:19.514 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:11.032 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:19.518 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:11.034 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:19.518 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:11.035 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:19.522 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:11.040 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:19.526 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:11.043 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:38.898 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:30.415 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:38.946 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:30.464 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:38.947 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:30.430 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:38.952 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:30.435 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:38.968 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:30.486 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:38.974 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:30.457 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:39.007 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:30.524 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:39.010 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:30.527 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:39.028 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:30.545 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:39.474 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:30.956 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:39.474 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:30.957 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:39.482 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:30.965 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:39.549 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:31.067 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:39.550 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:31.068 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:39.559 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:31.077 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:39.570 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:31.085 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:39.570 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:31.086 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:39.580 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:31.097 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:58.984 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:50.501 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:58.988 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:50.505 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.032 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:50.550 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.033 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:50.550 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.036 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:50.519 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.036 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:50.553 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.037 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:50.520 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.058 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:50.540 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.058 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:50.575 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.566 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:51.048 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.566 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:51.049 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.575 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:31:51.058 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.581 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:51.098 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.581 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:51.099 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.587 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:51.104 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.588 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:51.105 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.592 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:31:51.110 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:32:59.602 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:31:51.119 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.026 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:10.543 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.029 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:10.547 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.073 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:10.590 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.073 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:10.590 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.076 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:10.593 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.080 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:10.563 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.082 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:10.565 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.095 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:10.611 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.102 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:10.585 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.621 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:11.104 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.624 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:11.141 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.625 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:11.108 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.625 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:11.142 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.625 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:11.143 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.627 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:11.143 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.633 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:11.116 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.633 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:11.151 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:19.634 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:11.151 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.086 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:30.603 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.090 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:30.608 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.111 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:30.629 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.133 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:30.650 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.136 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:30.619 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.137 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:30.654 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.139 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:30.622 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.156 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:30.673 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.158 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:30.641 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.746 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:31.229 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.747 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:31.230 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.747 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:31.265 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.749 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:31.266 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.756 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:31.239 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.760 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:31.278 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.774 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:31.290 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.774 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:31.291 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:39.784 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:31.301 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:41.366 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:32.884 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:41.370 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:32.888 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:41.375 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:32.892 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:41.377 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:32.860 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:41.378 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:32.895 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:41.379 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:32.862 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:41.390 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:32.908 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:41.395 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:32.878 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:41.398 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:32.915 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:54.116 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:45.633 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:54.117 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:45.635 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:54.141 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:45.659 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:54.165 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:45.682 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:54.166 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:45.648 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:54.167 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:45.684 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:54.167 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:45.650 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:54.191 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:45.674 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:54.192 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:45.709 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:58.914 INFO:tasks.workunit.client.0.smithi106.stdout: ec_io: Running main() from gmock_main.cc 2024-04-04T21:33:58.914 INFO:tasks.workunit.client.0.smithi106.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-04-04T21:33:58.914 INFO:tasks.workunit.client.0.smithi106.stdout: ec_io: [----------] Global test environment set-up. 2024-04-04T21:33:58.914 INFO:tasks.workunit.client.0.smithi106.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-04-04T21:33:58.914 INFO:tasks.workunit.client.0.smithi106.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-04-04T21:33:58.914 INFO:tasks.workunit.client.0.smithi106.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (11781 ms) 2024-04-04T21:33:58.914 INFO:tasks.workunit.client.0.smithi106.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-04-04T21:33:58.915 INFO:tasks.workunit.client.0.smithi106.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (1812984 ms) 2024-04-04T21:33:58.915 INFO:tasks.workunit.client.0.smithi106.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (1824765 ms total) 2024-04-04T21:33:58.915 INFO:tasks.workunit.client.0.smithi106.stdout: ec_io: 2024-04-04T21:33:58.915 INFO:tasks.workunit.client.0.smithi106.stdout: ec_io: [----------] Global test environment tear-down 2024-04-04T21:33:58.915 INFO:tasks.workunit.client.0.smithi106.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (1824765 ms total) 2024-04-04T21:33:58.915 INFO:tasks.workunit.client.0.smithi106.stdout: ec_io: [ PASSED ] 2 tests. 2024-04-04T21:33:59.312 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:50.829 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:59.375 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:50.892 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:59.375 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:50.858 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:59.378 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:50.860 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:59.378 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:50.895 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:59.380 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:50.897 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:59.399 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:32:50.882 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:59.399 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:32:50.916 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:33:59.419 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:32:50.937 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:04.253 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-04-04T21:34:04.254 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-04-04T21:34:04.254 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-04-04T21:34:04.254 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-04-04T21:34:04.254 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-04-04T21:34:04.254 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: notify 2024-04-04T21:34:04.254 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1808248 ms) 2024-04-04T21:34:04.254 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-04-04T21:34:04.254 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (368 ms) 2024-04-04T21:34:04.254 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1808616 ms total) 2024-04-04T21:34:04.254 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: 2024-04-04T21:34:04.254 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-04-04T21:34:04.255 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-04-04T21:34:04.255 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: notify 2024-04-04T21:34:04.255 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (837 ms) 2024-04-04T21:34:04.255 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-04-04T21:34:04.255 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: notify 2024-04-04T21:34:04.255 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (5464 ms) 2024-04-04T21:34:04.255 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-04-04T21:34:04.255 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (102 ms) 2024-04-04T21:34:04.255 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-04-04T21:34:04.255 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (103 ms) 2024-04-04T21:34:04.255 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-04-04T21:34:04.255 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: handle_notify cookie 94324862038080 notify_id 3852585664514 notifier_gid 21451 2024-04-04T21:34:04.255 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (123 ms) 2024-04-04T21:34:04.255 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-04-04T21:34:04.256 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: handle_notify cookie 94324862038080 notify_id 3856880631809 notifier_gid 21451 2024-04-04T21:34:04.256 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (517 ms) 2024-04-04T21:34:04.256 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-04-04T21:34:04.256 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: handle_notify cookie 94324862733184 notify_id 3856880631808 notifier_gid 21451 2024-04-04T21:34:04.256 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (500 ms) 2024-04-04T21:34:04.256 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-04-04T21:34:04.256 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: handle_notify cookie 94324862733184 notify_id 3861175599105 notifier_gid 21451 2024-04-04T21:34:04.256 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (377 ms) 2024-04-04T21:34:04.256 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-04-04T21:34:04.256 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: handle_notify cookie 94324862733184 notify_id 3861175599106 notifier_gid 21451 2024-04-04T21:34:04.256 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (121 ms) 2024-04-04T21:34:04.256 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-04-04T21:34:04.256 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: handle_notify cookie 94324862733184 notify_id 3861175599106 notifier_gid 21451 2024-04-04T21:34:04.256 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (117 ms) 2024-04-04T21:34:04.257 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-04-04T21:34:04.257 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: trying... 2024-04-04T21:34:04.257 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: handle_notify cookie 94324862745392 notify_id 3861175599105 notifier_gid 21451 2024-04-04T21:34:04.257 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: timed out 2024-04-04T21:34:09.209 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:00.726 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:09.293 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:00.810 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:09.341 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:00.824 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:09.343 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:00.861 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:09.345 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:00.828 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:09.356 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:00.873 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:09.361 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:00.878 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:09.381 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:00.864 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:09.399 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:00.915 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:19.321 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:10.804 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:19.323 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:10.806 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:19.324 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:10.841 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:19.342 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:10.860 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:19.353 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:10.836 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:19.394 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:10.911 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:19.408 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:10.925 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:19.411 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:10.928 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:19.450 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:10.967 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:22.036 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: flushing 2024-04-04T21:34:22.036 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: flushed 2024-04-04T21:34:22.036 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3091 ms) 2024-04-04T21:34:22.036 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-04-04T21:34:22.036 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: trying... 2024-04-04T21:34:22.036 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: handle_notify cookie 94324862745216 notify_id 3878355468291 notifier_gid 21451 2024-04-04T21:34:22.036 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: timed out 2024-04-04T21:34:22.036 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: flushing 2024-04-04T21:34:22.036 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: flushed 2024-04-04T21:34:22.036 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3548 ms) 2024-04-04T21:34:22.036 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-04-04T21:34:22.037 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: List watches 2024-04-04T21:34:22.037 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: notify2 2024-04-04T21:34:22.037 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: handle_notify cookie 94324862745216 notify_id 3891240370180 notifier_gid 21451 2024-04-04T21:34:22.037 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: notify2 done 2024-04-04T21:34:22.037 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: watch_check 2024-04-04T21:34:22.037 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: unwatch2 2024-04-04T21:34:22.037 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: flushing 2024-04-04T21:34:22.037 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: done 2024-04-04T21:34:22.037 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3867 ms) 2024-04-04T21:34:22.037 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-04-04T21:34:22.037 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: List watches 2024-04-04T21:34:22.037 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: notify2 2024-04-04T21:34:22.038 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: handle_notify cookie 94324862745216 notify_id 3904125272067 notifier_gid 21451 2024-04-04T21:34:22.038 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: notify2 done 2024-04-04T21:34:22.038 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: watch_check 2024-04-04T21:34:22.038 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: unwatch2 2024-04-04T21:34:22.038 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: flushing 2024-04-04T21:34:22.038 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: done 2024-04-04T21:34:22.038 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3731 ms) 2024-04-04T21:34:22.038 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (22499 ms total) 2024-04-04T21:34:22.038 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: 2024-04-04T21:34:22.038 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-04-04T21:34:22.038 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (1848117 ms total) 2024-04-04T21:34:22.038 INFO:tasks.workunit.client.0.smithi106.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-04-04T21:34:22.039 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:34:22.039 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42022 2024-04-04T21:34:22.039 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 42022 2024-04-04T21:34:22.039 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:34:22.039 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42048 2024-04-04T21:34:22.039 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 42048 2024-04-04T21:34:22.039 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:34:22.040 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41275 2024-04-04T21:34:22.040 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41275 2024-04-04T21:34:22.040 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:34:22.040 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41910 2024-04-04T21:34:22.040 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41910 2024-04-04T21:34:22.040 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:34:22.040 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41770 2024-04-04T21:34:22.040 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41770 2024-04-04T21:34:22.040 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:34:22.040 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41139 2024-04-04T21:34:22.040 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41139 2024-04-04T21:34:22.040 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:34:22.040 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41206 2024-04-04T21:34:22.041 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41206 2024-04-04T21:34:22.041 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:34:22.041 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41813 2024-04-04T21:34:22.041 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41813 2024-04-04T21:34:22.041 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:34:22.041 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41129 2024-04-04T21:34:22.041 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41129 2024-04-04T21:34:24.592 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:16.109 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:24.604 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:16.122 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:24.633 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:16.151 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:24.648 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:16.130 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:24.651 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:16.134 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:24.669 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:16.151 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:24.671 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:16.188 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:24.675 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:16.192 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:24.696 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:16.213 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:39.650 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:31.167 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:39.720 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:31.238 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:39.780 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:31.298 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:39.785 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:31.302 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:39.787 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:31.304 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:39.801 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:31.283 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:39.810 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:31.293 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:39.831 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:31.348 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:39.856 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:31.339 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:44.560 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:36.077 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:44.561 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:36.078 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:44.580 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:36.097 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:44.615 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:36.098 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:44.616 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:36.099 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:44.628 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:36.145 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:44.633 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:36.150 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:44.633 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:36.116 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:44.659 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:36.176 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:54.662 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:46.179 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:54.720 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:46.238 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:54.724 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:46.207 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:54.736 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:46.218 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:54.745 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:46.263 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:54.763 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:46.280 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:54.766 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:46.249 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:54.769 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:46.286 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:54.797 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:46.314 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:59.598 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:51.115 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:59.599 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:51.117 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:59.615 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:33:51.132 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:59.645 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:51.127 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:59.645 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:51.128 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:59.653 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:51.170 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:59.654 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:51.171 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:59.659 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:33:51.142 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:34:59.668 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:33:51.185 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:09.666 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:01.183 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:09.723 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:01.240 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:09.725 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:01.207 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:09.727 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:01.210 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:09.748 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:01.265 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:09.751 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:01.234 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:09.784 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:01.301 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:09.786 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:01.303 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:09.812 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:01.328 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:13.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: Running main() from gmock_main.cc 2024-04-04T21:35:13.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-04-04T21:35:13.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [----------] Global test environment set-up. 2024-04-04T21:35:13.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-04-04T21:35:13.735 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.TooBig (6951 ms) 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (1804823 ms) 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3343 ms) 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3318 ms) 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3749 ms) 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3038 ms) 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (2653 ms) 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-04-04T21:35:13.736 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3706 ms) 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (5651 ms) 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3273 ms) 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (4397 ms) 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (4465 ms) 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (4777 ms) 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3251 ms) 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.Flush (3495 ms) 2024-04-04T21:35:13.737 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-04-04T21:35:13.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3346 ms) 2024-04-04T21:35:13.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-04-04T21:35:13.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2678 ms) 2024-04-04T21:35:13.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-04-04T21:35:13.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3376 ms) 2024-04-04T21:35:13.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-04-04T21:35:13.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3613 ms) 2024-04-04T21:35:13.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-04-04T21:35:13.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3042 ms) 2024-04-04T21:35:13.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-04-04T21:35:13.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3109 ms) 2024-04-04T21:35:13.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-04-04T21:35:13.738 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3969 ms) 2024-04-04T21:35:13.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-04-04T21:35:13.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3436 ms) 2024-04-04T21:35:13.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-04-04T21:35:13.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3426 ms) 2024-04-04T21:35:13.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-04-04T21:35:13.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (4442 ms) 2024-04-04T21:35:13.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-04-04T21:35:13.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (4704 ms) 2024-04-04T21:35:13.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [----------] 26 tests from LibRadosAio (1900031 ms total) 2024-04-04T21:35:13.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: 2024-04-04T21:35:13.739 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-04-04T21:35:19.963 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:11.480 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:19.977 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:11.493 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:20.018 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:11.533 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:20.019 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:11.501 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:20.021 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:11.504 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:20.043 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:11.560 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:20.046 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:11.563 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:20.047 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:11.530 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:20.083 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:11.600 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:24.821 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:16.338 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:24.824 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:16.342 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:24.839 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:16.356 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:24.871 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:16.353 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:24.871 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:16.388 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:24.871 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:16.354 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:24.875 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:16.392 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:24.886 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:16.369 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:24.890 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:16.406 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:40.089 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:31.572 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:40.090 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:31.573 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:40.093 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:31.610 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:40.097 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:31.614 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:40.147 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:31.629 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:40.163 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:31.679 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:40.165 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:31.682 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:40.189 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:31.706 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:40.221 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:31.738 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:41.062 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:32.545 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:41.064 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:32.547 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:41.072 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:32.589 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:41.074 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:32.591 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:41.078 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:32.561 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:41.091 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:32.608 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:41.349 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:32.866 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:41.350 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:32.868 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:41.371 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:32.888 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:44.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWr api_aio_pp: Running main() from gmock_main.cc 2024-04-04T21:35:44.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-04-04T21:35:44.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-04-04T21:35:44.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-04-04T21:35:44.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-04-04T21:35:44.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (6937 ms) 2024-04-04T21:35:44.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-04-04T21:35:44.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (1827757 ms) 2024-04-04T21:35:44.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-04-04T21:35:44.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (8031 ms) 2024-04-04T21:35:44.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-04-04T21:35:44.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (4471 ms) 2024-04-04T21:35:44.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-04-04T21:35:44.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (4598 ms) 2024-04-04T21:35:44.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-04-04T21:35:44.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3425 ms) 2024-04-04T21:35:44.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-04-04T21:35:44.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (4406 ms) 2024-04-04T21:35:44.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-04-04T21:35:44.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3468 ms) 2024-04-04T21:35:44.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-04-04T21:35:44.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3807 ms) 2024-04-04T21:35:44.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-04-04T21:35:44.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3398 ms) 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3572 ms) 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3068 ms) 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (5020 ms) 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (4503 ms) 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3295 ms) 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3524 ms) 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (4502 ms) 2024-04-04T21:35:44.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-04-04T21:35:44.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (4370 ms) 2024-04-04T21:35:44.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-04-04T21:35:44.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3385 ms) 2024-04-04T21:35:44.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-04-04T21:35:44.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2648 ms) 2024-04-04T21:35:44.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-04-04T21:35:44.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3381 ms) 2024-04-04T21:35:44.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-04-04T21:35:44.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3061 ms) 2024-04-04T21:35:44.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-04-04T21:35:44.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2553 ms) 2024-04-04T21:35:44.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-04-04T21:35:44.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3378 ms) 2024-04-04T21:35:44.322 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-04-04T21:35:44.322 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3312 ms) 2024-04-04T21:35:44.322 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-04-04T21:35:44.322 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3650 ms) 2024-04-04T21:35:44.322 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-04-04T21:35:55.147 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:46.665 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:55.156 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:46.674 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:55.191 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:46.709 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:55.206 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:46.723 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:55.212 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:46.729 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:55.224 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:46.707 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:55.230 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:46.712 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:55.252 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:46.769 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:35:55.264 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:46.747 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:00.007 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:51.524 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:00.009 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:51.527 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:00.010 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:51.493 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:00.014 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:51.497 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:00.029 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:34:51.546 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:00.030 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:34:51.512 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:00.058 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:51.575 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:00.059 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:51.576 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:00.074 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:34:51.591 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:10.212 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:01.729 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:10.263 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:01.780 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:10.312 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:01.794 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:10.335 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:01.852 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:10.337 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:01.854 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:10.339 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:01.821 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:10.352 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:01.869 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:10.386 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:01.869 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:10.409 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:01.925 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:25.120 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:16.636 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:25.123 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:16.640 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:25.158 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:16.675 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:25.166 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:16.649 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:25.168 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:16.651 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:25.170 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:16.687 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:25.171 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:16.688 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:25.200 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:16.683 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:25.206 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:16.723 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:40.171 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:31.688 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:40.176 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:31.693 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:40.211 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:31.728 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:40.224 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:31.706 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:40.225 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:31.708 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:40.230 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:31.747 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:40.234 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:31.751 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:40.257 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:31.739 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:40.268 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:31.784 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:41.022 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:32.504 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:41.023 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:32.506 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:41.033 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:32.515 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:41.154 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:32.671 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:41.155 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:32.672 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:41.165 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:32.682 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:41.242 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:32.760 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:41.244 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:32.761 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:41.253 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:32.770 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:55.397 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:46.914 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:55.405 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:46.920 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:55.434 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:46.951 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:55.450 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:46.933 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:55.453 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:46.935 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:55.453 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:46.970 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:55.458 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:46.975 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:55.483 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:46.965 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:36:55.496 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:47.013 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:00.244 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:51.761 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:00.248 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:51.766 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:00.263 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:35:51.780 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:00.293 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:51.810 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:00.296 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:51.812 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:00.296 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:51.778 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:00.296 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:51.779 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:00.311 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:35:51.827 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:00.311 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:35:51.794 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:10.284 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:01.801 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:10.291 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:01.808 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:10.332 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:01.849 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:10.332 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:01.815 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:10.333 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:01.816 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:10.357 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:01.840 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:10.366 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:01.880 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:10.366 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:01.881 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:10.401 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:01.917 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:14.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAite 2024-04-04T21:37:14.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7180 ms) 2024-04-04T21:37:14.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-04-04T21:37:14.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (8431 ms) 2024-04-04T21:37:14.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-04-04T21:37:14.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7116 ms) 2024-04-04T21:37:14.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-04-04T21:37:14.022 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7875 ms) 2024-04-04T21:37:14.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-04-04T21:37:14.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7876 ms) 2024-04-04T21:37:14.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-04-04T21:37:14.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (8598 ms) 2024-04-04T21:37:14.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-04-04T21:37:14.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7787 ms) 2024-04-04T21:37:14.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-04-04T21:37:14.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7057 ms) 2024-04-04T21:37:14.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-04-04T21:37:14.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7598 ms) 2024-04-04T21:37:14.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-04-04T21:37:14.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (8070 ms) 2024-04-04T21:37:14.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-04-04T21:37:14.023 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7043 ms) 2024-04-04T21:37:14.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-04-04T21:37:14.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6885 ms) 2024-04-04T21:37:14.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-04-04T21:37:14.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7418 ms) 2024-04-04T21:37:14.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-04-04T21:37:14.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6923 ms) 2024-04-04T21:37:14.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-04-04T21:37:14.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6763 ms) 2024-04-04T21:37:14.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-04-04T21:37:14.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7664 ms) 2024-04-04T21:37:14.024 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (120284 ms total) 2024-04-04T21:37:14.025 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: 2024-04-04T21:37:14.025 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [----------] Global test environment tear-down 2024-04-04T21:37:14.025 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (2020315 ms total) 2024-04-04T21:37:14.025 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio: [ PASSED ] 42 tests. 2024-04-04T21:37:20.330 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:11.847 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:20.353 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:11.870 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:20.378 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:11.895 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:20.381 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:11.897 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:20.383 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:11.865 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:20.384 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:11.900 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:20.384 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:11.867 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:20.407 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:11.890 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:20.438 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:11.954 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:30.601 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:22.117 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:30.605 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:22.123 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:30.628 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:22.146 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:30.651 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:22.133 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:30.654 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:22.137 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:30.656 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:22.173 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:30.658 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:22.175 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:30.677 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:22.159 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:30.688 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:22.205 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:38.241 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:29.758 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:38.245 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:29.763 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:38.276 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:29.793 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:38.293 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:29.776 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:38.297 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:29.780 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:38.319 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:29.835 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:38.319 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:29.802 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:38.320 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:29.837 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:38.342 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:29.859 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:40.595 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:32.077 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:40.595 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:32.113 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:40.596 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:32.078 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:40.599 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:32.117 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:40.609 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:32.091 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:40.617 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:32.134 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:40.653 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:32.169 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:40.654 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:32.171 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:40.664 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:32.181 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:45.473 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:36.990 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:45.475 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:36.992 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:45.488 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:37.006 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:45.520 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:37.002 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:45.520 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:37.037 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:45.520 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:37.003 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:45.522 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:37.039 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:45.533 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:37.016 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:37:45.542 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:37.058 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:00.694 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:52.211 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:00.698 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:52.215 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:00.727 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:36:52.245 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:00.744 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:52.226 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:00.744 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:52.227 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:00.746 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:52.263 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:00.750 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:52.266 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:00.773 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:36:52.255 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:00.778 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:36:52.295 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:15.650 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:07.166 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:15.651 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:07.169 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:15.678 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:07.196 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:15.697 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:07.180 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:15.698 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:07.181 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:15.700 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:07.216 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:15.706 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:07.220 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:15.726 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:07.208 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:15.729 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:07.246 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:18.318 INFO:tasks.workunit.client.0.smithi106.stdout:io.RoundTripAppendPP (3086 ms) 2024-04-04T21:38:18.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-04-04T21:38:18.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3578 ms) 2024-04-04T21:38:18.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-04-04T21:38:18.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3162 ms) 2024-04-04T21:38:18.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-04-04T21:38:18.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3192 ms) 2024-04-04T21:38:18.318 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-04-04T21:38:18.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: setting pool EIO 2024-04-04T21:38:18.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: max_success 67, min_failed 68 2024-04-04T21:38:18.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (5396 ms) 2024-04-04T21:38:18.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-04-04T21:38:18.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4414 ms) 2024-04-04T21:38:18.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (1950349 ms total) 2024-04-04T21:38:18.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: 2024-04-04T21:38:18.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-04-04T21:38:18.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-04-04T21:38:18.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3414 ms) 2024-04-04T21:38:18.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-04-04T21:38:18.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (10241 ms) 2024-04-04T21:38:18.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-04-04T21:38:18.319 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (16741 ms) 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (2765 ms) 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (33161 ms total) 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3684 ms) 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3684 ms total) 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14369 ms) 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7587 ms) 2024-04-04T21:38:18.320 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-04-04T21:38:18.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6872 ms) 2024-04-04T21:38:18.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-04-04T21:38:18.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6641 ms) 2024-04-04T21:38:18.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-04-04T21:38:18.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3412 ms) 2024-04-04T21:38:18.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-04-04T21:38:18.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7884 ms) 2024-04-04T21:38:18.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-04-04T21:38:18.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7191 ms) 2024-04-04T21:38:18.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-04-04T21:38:18.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6678 ms) 2024-04-04T21:38:18.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-04-04T21:38:18.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7804 ms) 2024-04-04T21:38:18.321 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-04-04T21:38:18.322 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6715 ms) 2024-04-04T21:38:18.322 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-04-04T21:38:18.322 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7750 ms) 2024-04-04T21:38:18.322 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-04-04T21:38:18.322 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6855 ms) 2024-04-04T21:38:18.322 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-04-04T21:38:18.322 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7655 ms) 2024-04-04T21:38:20.958 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:12.475 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:20.969 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:12.487 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:21.007 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:12.524 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:21.019 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:12.502 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:21.032 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:12.514 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:21.037 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:12.554 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:21.049 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:12.532 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:21.053 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:12.569 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:21.077 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:12.593 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:35.794 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:27.311 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:35.795 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:27.313 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:35.823 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:27.340 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:35.841 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:27.323 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:35.842 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:27.325 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:35.844 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:27.361 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:35.846 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:27.362 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:35.871 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:27.354 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:35.880 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:27.397 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:40.805 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:32.322 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:40.809 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:32.326 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:40.823 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:32.340 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:40.856 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:32.339 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:40.859 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:32.342 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:40.862 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:32.379 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:40.864 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:32.381 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:40.872 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:32.355 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:40.878 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:32.395 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:50.843 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:42.360 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:50.846 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:42.364 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:50.868 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:42.385 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:50.893 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:42.409 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:50.894 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:42.410 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:50.896 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:42.379 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:50.898 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:42.380 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:50.918 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:42.434 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:38:50.923 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:42.406 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:00.905 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:52.422 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:00.910 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:52.427 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:00.932 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:37:52.449 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:00.955 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:52.438 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:00.956 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:52.473 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:00.957 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:52.440 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:00.958 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:52.475 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:00.979 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:37:52.462 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:00.980 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:37:52.497 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:10.385 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-04-04T21:39:10.386 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3525 ms) 2024-04-04T21:39:10.386 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-04-04T21:39:10.386 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7834 ms) 2024-04-04T21:39:10.386 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-04-04T21:39:10.386 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (8071 ms) 2024-04-04T21:39:10.386 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-04-04T21:39:10.386 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7992 ms) 2024-04-04T21:39:10.390 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-04-04T21:39:10.390 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (8934 ms) 2024-04-04T21:39:10.390 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-04-04T21:39:10.390 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7950 ms) 2024-04-04T21:39:10.390 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-04-04T21:39:10.390 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7758 ms) 2024-04-04T21:39:10.390 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (149477 ms total) 2024-04-04T21:39:10.390 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: 2024-04-04T21:39:10.390 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-04-04T21:39:10.390 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (2136672 ms total) 2024-04-04T21:39:10.390 INFO:tasks.workunit.client.0.smithi106.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-04-04T21:39:10.392 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.392 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41150 2024-04-04T21:39:10.392 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41150 2024-04-04T21:39:10.392 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.392 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42126 2024-04-04T21:39:10.393 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 42126 2024-04-04T21:39:10.393 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.393 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41384 2024-04-04T21:39:10.393 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41384 2024-04-04T21:39:10.393 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.393 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42199 2024-04-04T21:39:10.393 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 42199 2024-04-04T21:39:10.393 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.393 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41441 2024-04-04T21:39:10.393 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41441 2024-04-04T21:39:10.393 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.393 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42000 2024-04-04T21:39:10.393 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 42000 2024-04-04T21:39:10.393 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42322 2024-04-04T21:39:10.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 42322 2024-04-04T21:39:10.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41407 2024-04-04T21:39:10.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41407 2024-04-04T21:39:10.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41830 2024-04-04T21:39:10.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41830 2024-04-04T21:39:10.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42105 2024-04-04T21:39:10.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 42105 2024-04-04T21:39:10.394 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41943 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41943 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42159 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 42159 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41763 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41763 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41320 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41320 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42270 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 42270 2024-04-04T21:39:10.395 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41643 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41643 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41122 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41122 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41185 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41185 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41569 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41569 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41224 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41224 2024-04-04T21:39:10.396 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41662 2024-04-04T21:39:10.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41662 2024-04-04T21:39:10.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=41724 2024-04-04T21:39:10.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 41724 2024-04-04T21:39:10.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42081 2024-04-04T21:39:10.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 42081 2024-04-04T21:39:10.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ for t in "${!pids[@]}" 2024-04-04T21:39:10.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ pid=42359 2024-04-04T21:39:10.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ wait 42359 2024-04-04T21:39:10.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ exit 0 2024-04-04T21:39:10.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ cleanup 2024-04-04T21:39:10.397 INFO:tasks.workunit.client.0.smithi106.stderr:+ pkill -P 41116 2024-04-04T21:39:10.412 INFO:tasks.workunit.client.0.smithi106.stderr:+ true 2024-04-04T21:39:10.413 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-04T21:39:10.413 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-04T21:39:10.441 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-04-04T21:39:10.441 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-04T21:39:10.751 DEBUG:teuthology.parallel:result is None 2024-04-04T21:39:10.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-04T21:39:10.778 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-04T21:39:10.778 DEBUG:teuthology.orchestra.run.smithi106:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-04T21:39:10.845 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-04T21:39:10.845 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-04T21:39:10.855 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-04T21:39:10.855 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-04T21:39:10.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.993+0000 7fd8dba27640 10 monclient: get_monmap_and_config 2024-04-04T21:39:10.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.993+0000 7fd8dba27640 10 monclient: build_initial_monmap 2024-04-04T21:39:10.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.993+0000 7fd8dba27640 10 monclient: monmap: 2024-04-04T21:39:10.994 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:10.994 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:10.994 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:10.994677+0000 2024-04-04T21:39:10.994 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:10.994677+0000 2024-04-04T21:39:10.994 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:10.994 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:10.994 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:10.994 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:10.994 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:10.994 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:10.995 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:10.995 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:10.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.994+0000 7fd8dba27640 1 Processor -- start 2024-04-04T21:39:10.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.994+0000 7fd8dba27640 1 -- start start 2024-04-04T21:39:10.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.994+0000 7fd8dba27640 10 monclient: init 2024-04-04T21:39:10.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.994+0000 7fd8dba27640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:10.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.994+0000 7fd8dba27640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-04T21:39:10.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.994+0000 7fd8dba27640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd8d4140750 0x7fd8d4140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:10.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.994+0000 7fd8dba27640 10 monclient(hunting): picked mon.noname-c con 0x7fd8d4140750 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:10.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.994+0000 7fd8dba27640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8d40aeaa0 addr v1:172.21.15.203:6789/0 2024-04-04T21:39:10.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.994+0000 7fd8dba27640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8d41440f0 0x7fd8d414af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:10.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.994+0000 7fd8dba27640 10 monclient(hunting): picked mon.noname-b con 0x7fd8d41440f0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:10.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.994+0000 7fd8dba27640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d4072990 con 0x7fd8d40aeaa0 2024-04-04T21:39:10.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.995+0000 7fd8dba27640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:10.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.995+0000 7fd8dba27640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d4140b30 con 0x7fd8d4140750 2024-04-04T21:39:10.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.995+0000 7fd8dba27640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:10.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.995+0000 7fd8dba27640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d4141410 con 0x7fd8d41440f0 2024-04-04T21:39:10.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.995+0000 7fd8dba27640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:10.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.995+0000 7fd8dba27640 10 monclient(hunting): authenticate will time out at 3600.262214s 2024-04-04T21:39:10.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.995+0000 7fd8d97c3640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd8d4140750 0x7fd8d4140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:11.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.995+0000 7fd8d9fc4640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8d41440f0 0x7fd8d414af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:11.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.995+0000 7fd8d97c3640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd8d4140750 0x7fd8d4140ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:45418/0 (socket says 172.21.15.106:45418) 2024-04-04T21:39:11.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.995+0000 7fd8d97c3640 1 -- 172.21.15.106:0/4289826856 learned_addr learned my addr 172.21.15.106:0/4289826856 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:11.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.995+0000 7fd8d97c3640 10 monclient(hunting): get_auth_request con 0x7fd8d4140750 auth_method 0 2024-04-04T21:39:11.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.995+0000 7fd8d97c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:11.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.995+0000 7fd8d97c3640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:11.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.995+0000 7fd8d97c3640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:11.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.996+0000 7fd8d9fc4640 10 monclient(hunting): get_auth_request con 0x7fd8d41440f0 auth_method 0 2024-04-04T21:39:11.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.996+0000 7fd8d9fc4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:11.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.996+0000 7fd8d9fc4640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:11.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.996+0000 7fd8d9fc4640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:11.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.996+0000 7fd8d9fc4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:11.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.996+0000 7fd8d9fc4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:11.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.996+0000 7fd8d9fc4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:11.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.996+0000 7fd8d97c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:11.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.996+0000 7fd8d97c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:11.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.996+0000 7fd8d97c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:11.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.998+0000 7fd8d97c3640 10 monclient(hunting): handle_auth_done global_id 22027 payload 274 2024-04-04T21:39:11.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.998+0000 7fd8d97c3640 1 -- 172.21.15.106:0/4289826856 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8d41440f0 msgr2=0x7fd8d414af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:11.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.998+0000 7fd8d97c3640 1 --2- 172.21.15.106:0/4289826856 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8d41440f0 0x7fd8d414af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.998+0000 7fd8d97c3640 1 -- 172.21.15.106:0/4289826856 >> v1:172.21.15.203:6789/0 conn(0x7fd8d40aeaa0 legacy=0x7fd8d41439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:11.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.998+0000 7fd8d97c3640 10 monclient: _finish_hunting 0 2024-04-04T21:39:11.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.998+0000 7fd8d97c3640 1 monclient: found mon.noname-c 2024-04-04T21:39:11.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.998+0000 7fd8d97c3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:11.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.998+0000 7fd8d97c3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:39:11.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:10.998+0000 7fd8d97c3640 1 -- 172.21.15.106:0/4289826856 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d40707e0 con 0x7fd8d4140750 2024-04-04T21:39:11.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.001+0000 7fd8d97c3640 1 --2- 172.21.15.106:0/4289826856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d4140750 0x7fd8d4140ed0 secure :-1 s=READY pgs=5526 cs=0 l=1 rev1=1 crypto rx=0x7fd8d00098a0 tx=0x7fd8d002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53f52ed73bd92d83 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:11.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8cbfff640 1 -- 172.21.15.106:0/4289826856 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0035020 con 0x7fd8d4140750 2024-04-04T21:39:11.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:11.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8cbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:39:11.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8cbfff640 10 monclient: dump: 2024-04-04T21:39:11.007 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:11.007 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:11.007 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.007 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.007 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:11.007 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:11.007 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:11.008 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:11.008 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:11.008 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:11.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8cbfff640 10 monclient: _finish_auth 0 2024-04-04T21:39:11.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8cbfff640 10 monclient: _check_auth_tickets 2024-04-04T21:39:11.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:11.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8cbfff640 1 -- 172.21.15.106:0/4289826856 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d0031640 con 0x7fd8d4140750 2024-04-04T21:39:11.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8cbfff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:11.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8cbfff640 1 -- 172.21.15.106:0/4289826856 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d002d040 con 0x7fd8d4140750 2024-04-04T21:39:11.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8dba27640 5 monclient: authenticate success, global_id 22027 2024-04-04T21:39:11.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8dba27640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:11.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8dba27640 4 set_mon_vals no callback set 2024-04-04T21:39:11.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:11.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:11.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8cbfff640 10 monclient: dump: 2024-04-04T21:39:11.009 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:11.009 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:11.009 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.009 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.009 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:11.009 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:11.009 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:11.009 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:11.009 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:11.009 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:11.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8dba27640 10 monclient: shutdown 2024-04-04T21:39:11.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8dba27640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:11.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8dba27640 1 -- 172.21.15.106:0/4289826856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d4140750 msgr2=0x7fd8d4140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:11.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.006+0000 7fd8dba27640 1 --2- 172.21.15.106:0/4289826856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d4140750 0x7fd8d4140ed0 secure :-1 s=READY pgs=5526 cs=0 l=1 rev1=1 crypto rx=0x7fd8d00098a0 tx=0x7fd8d002b8f0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.007+0000 7fd8dba27640 1 -- 172.21.15.106:0/4289826856 shutdown_connections 2024-04-04T21:39:11.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.007+0000 7fd8dba27640 1 --2- 172.21.15.106:0/4289826856 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8d41440f0 0x7fd8d414af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.007+0000 7fd8dba27640 1 --2- 172.21.15.106:0/4289826856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d4140750 0x7fd8d4140ed0 unknown :-1 s=CLOSED pgs=5526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.007+0000 7fd8dba27640 1 -- 172.21.15.106:0/4289826856 >> 172.21.15.106:0/4289826856 conn(0x7fd8d40b3fb0 msgr2=0x7fd8d40b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:11.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.007+0000 7fd8dba27640 1 -- 172.21.15.106:0/4289826856 shutdown_connections 2024-04-04T21:39:11.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.007+0000 7fd8dba27640 1 -- 172.21.15.106:0/4289826856 wait complete. 2024-04-04T21:39:11.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.007+0000 7fd8dba27640 10 monclient: build_initial_monmap 2024-04-04T21:39:11.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.007+0000 7fd8dba27640 10 monclient: monmap: 2024-04-04T21:39:11.014 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:11.014 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:11.014 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:11.014 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:11.014 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:11.014 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:11.014 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:11.014 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:11.014 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:11.014 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:11.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.007+0000 7fd8dba27640 1 librados: starting msgr at 2024-04-04T21:39:11.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.007+0000 7fd8dba27640 1 librados: starting objecter 2024-04-04T21:39:11.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 1 Processor -- start 2024-04-04T21:39:11.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 1 -- start start 2024-04-04T21:39:11.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 1 librados: setting wanted keys 2024-04-04T21:39:11.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 1 librados: calling monclient init 2024-04-04T21:39:11.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 10 monclient: init 2024-04-04T21:39:11.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:11.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-04T21:39:11.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d40aeaa0 0x7fd8d406f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:11.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 10 monclient(hunting): picked mon.noname-b con 0x7fd8d40aeaa0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:11.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8d4140750 0x7fd8d413d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:11.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 10 monclient(hunting): picked mon.noname-a con 0x7fd8d4140750 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:11.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8d41440f0 0x7fd8d406f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:11.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 10 monclient(hunting): picked mon.noname-c con 0x7fd8d41440f0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40af090 con 0x7fd8d4140750 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40aef10 con 0x7fd8d40aeaa0 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d4071d80 con 0x7fd8d41440f0 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.008+0000 7fd8dba27640 10 monclient(hunting): authenticate will time out at 3600.275874s 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d97c3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d40aeaa0 0x7fd8d406f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d9fc4640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8d41440f0 0x7fd8d406f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d97c3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d40aeaa0 0x7fd8d406f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:45426/0 (socket says 172.21.15.106:45426) 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d97c3640 1 -- 172.21.15.106:0/809875930 learned_addr learned my addr 172.21.15.106:0/809875930 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d97c3640 10 monclient(hunting): get_auth_request con 0x7fd8d40aeaa0 auth_method 0 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d97c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:11.016 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d97c3640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:11.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d97c3640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:11.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d9fc4640 10 monclient(hunting): get_auth_request con 0x7fd8d41440f0 auth_method 0 2024-04-04T21:39:11.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d9fc4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:11.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d9fc4640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:11.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d9fc4640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:11.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d97c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:11.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d97c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:11.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.009+0000 7fd8d97c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:11.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.010+0000 7fd8d9fc4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:11.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.010+0000 7fd8d9fc4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:11.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.010+0000 7fd8d9fc4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:11.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.012+0000 7fd8d97c3640 10 monclient(hunting): handle_auth_done global_id 22030 payload 824 2024-04-04T21:39:11.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.012+0000 7fd8d97c3640 1 -- 172.21.15.106:0/809875930 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8d41440f0 msgr2=0x7fd8d406f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:11.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.012+0000 7fd8d97c3640 1 --2- 172.21.15.106:0/809875930 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8d41440f0 0x7fd8d406f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.012+0000 7fd8d97c3640 1 -- 172.21.15.106:0/809875930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8d4140750 msgr2=0x7fd8d413d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:11.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.012+0000 7fd8d97c3640 1 --2- 172.21.15.106:0/809875930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8d4140750 0x7fd8d413d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.012+0000 7fd8d97c3640 10 monclient: _finish_hunting 0 2024-04-04T21:39:11.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.012+0000 7fd8d97c3640 1 monclient: found mon.noname-b 2024-04-04T21:39:11.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.012+0000 7fd8d97c3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:11.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.012+0000 7fd8d97c3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:11.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.012+0000 7fd8d97c3640 1 -- 172.21.15.106:0/809875930 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d4142880 con 0x7fd8d40aeaa0 2024-04-04T21:39:11.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.014+0000 7fd8d97c3640 1 --2- 172.21.15.106:0/809875930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d40aeaa0 0x7fd8d406f100 secure :-1 s=READY pgs=5527 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0000c00 tx=0x7fd8d00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:11.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.014+0000 7fd8c9ffb640 10 client.?.objecter ms_handle_connect 0x7fd8d40aeaa0 2024-04-04T21:39:11.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.014+0000 7fd8c9ffb640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:11.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8c9ffb640 1 -- 172.21.15.106:0/809875930 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0040070 con 0x7fd8d40aeaa0 2024-04-04T21:39:11.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:11.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8c9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:11.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8c9ffb640 10 monclient: dump: 2024-04-04T21:39:11.020 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:11.020 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:11.020 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.020 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.020 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:11.020 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8c9ffb640 10 monclient: _finish_auth 0 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8c9ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8c9ffb640 1 -- 172.21.15.106:0/809875930 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d00399f0 con 0x7fd8d40aeaa0 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8c9ffb640 1 -- 172.21.15.106:0/809875930 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0035040 con 0x7fd8d40aeaa0 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8c9ffb640 10 monclient: dump: 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:11.021 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:11.022 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.022 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.022 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:11.022 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:11.022 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:11.022 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:11.022 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:11.022 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:11.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8dba27640 5 monclient: authenticate success, global_id 22030 2024-04-04T21:39:11.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8dba27640 10 monclient: _renew_subs 2024-04-04T21:39:11.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8dba27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:11.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8dba27640 1 -- 172.21.15.106:0/809875930 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d41394a0 con 0x7fd8d40aeaa0 2024-04-04T21:39:11.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8cb7fe640 4 set_mon_vals no callback set 2024-04-04T21:39:11.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8dba27640 10 client.22030.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:11.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8dba27640 10 monclient: _renew_subs 2024-04-04T21:39:11.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8dba27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:11.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8dba27640 1 -- 172.21.15.106:0/809875930 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d40ad830 con 0x7fd8d40aeaa0 2024-04-04T21:39:11.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.018+0000 7fd8dba27640 1 librados: init done 2024-04-04T21:39:11.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.019+0000 7fd8dba27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:11.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.019+0000 7fd8dba27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:11.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.019+0000 7fd8dba27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:11.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.019+0000 7fd8dba27640 1 -- 172.21.15.106:0/809875930 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d404f050 con 0x7fd8d40aeaa0 2024-04-04T21:39:11.032 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:02.549 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:11.035 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:02.552 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:11.059 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:02.576 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:11.083 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:02.600 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:11.084 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:02.601 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:11.106 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:02.623 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:11.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.311+0000 7fd8c9ffb640 1 -- 172.21.15.106:0/809875930 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd8d004b060 con 0x7fd8d40aeaa0 2024-04-04T21:39:11.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.312+0000 7fd8c9ffb640 1 --2- 172.21.15.106:0/809875930 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd8b806ceb0 0x7fd8b806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:11.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.312+0000 7fd8d8fc2640 1 --2- 172.21.15.106:0/809875930 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd8b806ceb0 0x7fd8b806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:11.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.312+0000 7fd8d8fc2640 10 monclient: get_auth_request con 0x7fd8b806ceb0 auth_method 0 2024-04-04T21:39:11.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.312+0000 7fd8d8fc2640 1 --2- 172.21.15.106:0/809875930 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd8b806ceb0 0x7fd8b806f370 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4000f60 tx=0x7fd8c40079b0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:11.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.312+0000 7fd8c9ffb640 10 client.22030.objecter ms_handle_connect 0x7fd8b806ceb0 2024-04-04T21:39:11.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.315+0000 7fd8c9ffb640 1 -- 172.21.15.106:0/809875930 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7fd8d0078760 con 0x7fd8d40aeaa0 2024-04-04T21:39:11.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.315+0000 7fd8c9ffb640 10 client.22030.objecter ms_dispatch 0x7fd8d4000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:11.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.315+0000 7fd8c9ffb640 3 client.22030.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:11.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.315+0000 7fd8c9ffb640 3 client.22030.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:11.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.315+0000 7fd8c9ffb640 20 client.22030.objecter dump_active .. 0 homeless 2024-04-04T21:39:11.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.325+0000 7fd8c9ffb640 1 -- 172.21.15.106:0/809875930 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd8d007adc0 con 0x7fd8d40aeaa0 2024-04-04T21:39:11.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.325+0000 7fd8c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:11.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.325+0000 7fd8c9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:11.377 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:02.859 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:11.378 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:02.861 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:11.400 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:02.883 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:11.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.522+0000 7fd8dba27640 1 -- 172.21.15.106:0/809875930 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd8d4071950 con 0x7fd8b806ceb0 2024-04-04T21:39:11.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.526+0000 7fd8c9ffb640 1 -- 172.21.15.106:0/809875930 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+116264 (secure 0 0 0) 0x7fd8d4071950 con 0x7fd8b806ceb0 2024-04-04T21:39:11.527 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:39:11.528 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-04-04T21:39:11.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 10 librados: watch_flush enter 2024-04-04T21:39:11.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 10 librados: watch_flush exit 2024-04-04T21:39:11.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 20 client.22030.objecter shutdown clearing up homeless session... 2024-04-04T21:39:11.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 10 client.22030.objecter successfully canceled tick 2024-04-04T21:39:11.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 1 -- 172.21.15.106:0/809875930 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd8b806ceb0 msgr2=0x7fd8b806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:11.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 1 --2- 172.21.15.106:0/809875930 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd8b806ceb0 0x7fd8b806f370 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4000f60 tx=0x7fd8c40079b0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 10 monclient: shutdown 2024-04-04T21:39:11.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:11.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 1 -- 172.21.15.106:0/809875930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d40aeaa0 msgr2=0x7fd8d406f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:11.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 1 --2- 172.21.15.106:0/809875930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d40aeaa0 0x7fd8d406f100 secure :-1 s=READY pgs=5527 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0000c00 tx=0x7fd8d00046c0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 1 -- 172.21.15.106:0/809875930 shutdown_connections 2024-04-04T21:39:11.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 1 --2- 172.21.15.106:0/809875930 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd8b806ceb0 0x7fd8b806f370 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 1 --2- 172.21.15.106:0/809875930 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8d41440f0 0x7fd8d406f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 1 --2- 172.21.15.106:0/809875930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8d4140750 0x7fd8d413d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 1 --2- 172.21.15.106:0/809875930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d40aeaa0 0x7fd8d406f100 unknown :-1 s=CLOSED pgs=5527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 1 -- 172.21.15.106:0/809875930 >> 172.21.15.106:0/809875930 conn(0x7fd8d40b3fb0 msgr2=0x7fd8d413a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:11.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 1 -- 172.21.15.106:0/809875930 shutdown_connections 2024-04-04T21:39:11.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 1 -- 172.21.15.106:0/809875930 wait complete. 2024-04-04T21:39:11.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.529+0000 7fd8dba27640 1 librados: shutdown 2024-04-04T21:39:11.541 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":2312,"stamp":"2024-04-04T21:39:11.059877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":87,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":568128,"kb_used_data":42376,"kb_used_omap":12,"kb_used_meta":525619,"kb_avail":754406592,"statfs":{"total":773094113280,"available":772512350208,"internally_reserved":0,"allocated":43393024,"data_stored":21620897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":538234417},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":158,"apply_latency_ms":158,"commit_latency_ns":158000000,"apply_latency_ns":158000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.996579"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2332,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.039530+0000","last_change":"2024-04-04T21:03:54.942652+0000","last_active":"2024-04-04T21:39:07.039530+0000","last_peered":"2024-04-04T21:39:07.039530+0000","last_clean":"2024-04-04T21:39:07.039530+0000","last_became_active":"2024-04-04T21:03:54.933965+0000","last_became_peered":"2024-04-04T21:03:54.933965+0000","last_unstale":"2024-04-04T21:39:07.039530+0000","last_undegraded":"2024-04-04T21:39:07.039530+0000","last_fullsized":"2024-04-04T21:39:07.039530+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:47:55.727002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2305,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.360894+0000","last_change":"2024-04-04T21:03:44.565986+0000","last_active":"2024-04-04T21:39:08.360894+0000","last_peered":"2024-04-04T21:39:08.360894+0000","last_clean":"2024-04-04T21:39:08.360894+0000","last_became_active":"2024-04-04T21:03:44.551107+0000","last_became_peered":"2024-04-04T21:03:44.551107+0000","last_unstale":"2024-04-04T21:39:08.360894+0000","last_undegraded":"2024-04-04T21:39:08.360894+0000","last_fullsized":"2024-04-04T21:39:08.360894+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:51:23.943936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2301,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.405327+0000","last_change":"2024-04-04T21:03:52.793808+0000","last_active":"2024-04-04T21:39:08.405327+0000","last_peered":"2024-04-04T21:39:08.405327+0000","last_clean":"2024-04-04T21:39:08.405327+0000","last_became_active":"2024-04-04T21:03:52.772128+0000","last_became_peered":"2024-04-04T21:03:52.772128+0000","last_unstale":"2024-04-04T21:39:08.405327+0000","last_undegraded":"2024-04-04T21:39:08.405327+0000","last_fullsized":"2024-04-04T21:39:08.405327+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:44:28.885034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2287,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.407788+0000","last_change":"2024-04-04T21:03:41.361551+0000","last_active":"2024-04-04T21:39:08.407788+0000","last_peered":"2024-04-04T21:39:08.407788+0000","last_clean":"2024-04-04T21:39:08.407788+0000","last_became_active":"2024-04-04T21:03:41.346161+0000","last_became_peered":"2024-04-04T21:03:41.346161+0000","last_unstale":"2024-04-04T21:39:08.407788+0000","last_undegraded":"2024-04-04T21:39:08.407788+0000","last_fullsized":"2024-04-04T21:39:08.407788+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:14:45.723197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2289,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.440013+0000","last_change":"2024-04-04T21:03:42.130175+0000","last_active":"2024-04-04T21:39:08.440013+0000","last_peered":"2024-04-04T21:39:08.440013+0000","last_clean":"2024-04-04T21:39:08.440013+0000","last_became_active":"2024-04-04T21:03:42.122180+0000","last_became_peered":"2024-04-04T21:03:42.122180+0000","last_unstale":"2024-04-04T21:39:08.440013+0000","last_undegraded":"2024-04-04T21:39:08.440013+0000","last_fullsized":"2024-04-04T21:39:08.440013+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:01:39.698809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2318,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.250973+0000","last_change":"2024-04-04T21:03:42.949631+0000","last_active":"2024-04-04T21:39:09.250973+0000","last_peered":"2024-04-04T21:39:09.250973+0000","last_clean":"2024-04-04T21:39:09.250973+0000","last_became_active":"2024-04-04T21:03:42.940728+0000","last_became_peered":"2024-04-04T21:03:42.940728+0000","last_unstale":"2024-04-04T21:39:09.250973+0000","last_undegraded":"2024-04-04T21:39:09.250973+0000","last_fullsized":"2024-04-04T21:39:09.250973+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:24:58.260325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2306,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.372867+0000","last_change":"2024-04-04T21:03:43.791130+0000","last_active":"2024-04-04T21:39:08.372867+0000","last_peered":"2024-04-04T21:39:08.372867+0000","last_clean":"2024-04-04T21:39:08.372867+0000","last_became_active":"2024-04-04T21:03:43.596620+0000","last_became_peered":"2024-04-04T21:03:43.596620+0000","last_unstale":"2024-04-04T21:39:08.372867+0000","last_undegraded":"2024-04-04T21:39:08.372867+0000","last_fullsized":"2024-04-04T21:39:08.372867+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T21:23:20.072459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2307,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.377536+0000","last_change":"2024-04-04T21:03:52.362151+0000","last_active":"2024-04-04T21:39:08.377536+0000","last_peered":"2024-04-04T21:39:08.377536+0000","last_clean":"2024-04-04T21:39:08.377536+0000","last_became_active":"2024-04-04T21:03:52.351392+0000","last_became_peered":"2024-04-04T21:03:52.351392+0000","last_unstale":"2024-04-04T21:39:08.377536+0000","last_undegraded":"2024-04-04T21:39:08.377536+0000","last_fullsized":"2024-04-04T21:39:08.377536+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T01:47:25.888369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"39'7","reported_seq":2332,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.239999+0000","last_change":"2024-04-04T21:03:50.565469+0000","last_active":"2024-04-04T21:39:09.239999+0000","last_peered":"2024-04-04T21:39:09.239999+0000","last_clean":"2024-04-04T21:39:09.239999+0000","last_became_active":"2024-04-04T21:03:50.287197+0000","last_became_peered":"2024-04-04T21:03:50.287197+0000","last_unstale":"2024-04-04T21:39:09.239999+0000","last_undegraded":"2024-04-04T21:39:09.239999+0000","last_fullsized":"2024-04-04T21:39:09.239999+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T04:15:05.649671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2290,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.366167+0000","last_change":"2024-04-04T21:03:44.924431+0000","last_active":"2024-04-04T21:39:08.366167+0000","last_peered":"2024-04-04T21:39:08.366167+0000","last_clean":"2024-04-04T21:39:08.366167+0000","last_became_active":"2024-04-04T21:03:44.913048+0000","last_became_peered":"2024-04-04T21:03:44.913048+0000","last_unstale":"2024-04-04T21:39:08.366167+0000","last_undegraded":"2024-04-04T21:39:08.366167+0000","last_fullsized":"2024-04-04T21:39:08.366167+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T22:52:04.454928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2286,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.112318+0000","last_change":"2024-04-04T21:03:44.496009+0000","last_active":"2024-04-04T21:39:07.112318+0000","last_peered":"2024-04-04T21:39:07.112318+0000","last_clean":"2024-04-04T21:39:07.112318+0000","last_became_active":"2024-04-04T21:03:44.451112+0000","last_became_peered":"2024-04-04T21:03:44.451112+0000","last_unstale":"2024-04-04T21:39:07.112318+0000","last_undegraded":"2024-04-04T21:39:07.112318+0000","last_fullsized":"2024-04-04T21:39:07.112318+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:12:30.730952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2334,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.024282+0000","last_change":"2024-04-04T21:03:44.379999+0000","last_active":"2024-04-04T21:39:07.024282+0000","last_peered":"2024-04-04T21:39:07.024282+0000","last_clean":"2024-04-04T21:39:07.024282+0000","last_became_active":"2024-04-04T21:03:44.366452+0000","last_became_peered":"2024-04-04T21:03:44.366452+0000","last_unstale":"2024-04-04T21:39:07.024282+0000","last_undegraded":"2024-04-04T21:39:07.024282+0000","last_fullsized":"2024-04-04T21:39:07.024282+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:21:39.463272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":2318,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.168312+0000","last_change":"2024-04-04T21:02:20.509950+0000","last_active":"2024-04-04T21:39:08.168312+0000","last_peered":"2024-04-04T21:39:08.168312+0000","last_clean":"2024-04-04T21:39:08.168312+0000","last_became_active":"2024-04-04T21:02:19.876220+0000","last_became_peered":"2024-04-04T21:02:19.876220+0000","last_unstale":"2024-04-04T21:39:08.168312+0000","last_undegraded":"2024-04-04T21:39:08.168312+0000","last_fullsized":"2024-04-04T21:39:08.168312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:05.463818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:05.463818+0000","last_clean_scrub_stamp":"2024-04-04T21:02:05.463818+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:18:20.734360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2323,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.257109+0000","last_change":"2024-04-04T21:02:24.476744+0000","last_active":"2024-04-04T21:39:09.257109+0000","last_peered":"2024-04-04T21:39:09.257109+0000","last_clean":"2024-04-04T21:39:09.257109+0000","last_became_active":"2024-04-04T21:02:24.466203+0000","last_became_peered":"2024-04-04T21:02:24.466203+0000","last_unstale":"2024-04-04T21:39:09.257109+0000","last_undegraded":"2024-04-04T21:39:09.257109+0000","last_fullsized":"2024-04-04T21:39:09.257109+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T21:40:50.021063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2289,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.369121+0000","last_change":"2024-04-04T21:03:40.772628+0000","last_active":"2024-04-04T21:39:08.369121+0000","last_peered":"2024-04-04T21:39:08.369121+0000","last_clean":"2024-04-04T21:39:08.369121+0000","last_became_active":"2024-04-04T21:03:40.758544+0000","last_became_peered":"2024-04-04T21:03:40.758544+0000","last_unstale":"2024-04-04T21:39:08.369121+0000","last_undegraded":"2024-04-04T21:39:08.369121+0000","last_fullsized":"2024-04-04T21:39:08.369121+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:22:11.238559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2324,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.235421+0000","last_change":"2024-04-04T21:02:24.174900+0000","last_active":"2024-04-04T21:39:09.235421+0000","last_peered":"2024-04-04T21:39:09.235421+0000","last_clean":"2024-04-04T21:39:09.235421+0000","last_became_active":"2024-04-04T21:02:24.146676+0000","last_became_peered":"2024-04-04T21:02:24.146676+0000","last_unstale":"2024-04-04T21:39:09.235421+0000","last_undegraded":"2024-04-04T21:39:09.235421+0000","last_fullsized":"2024-04-04T21:39:09.235421+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:47:28.827956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2287,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.411575+0000","last_change":"2024-04-04T21:03:39.388042+0000","last_active":"2024-04-04T21:39:08.411575+0000","last_peered":"2024-04-04T21:39:08.411575+0000","last_clean":"2024-04-04T21:39:08.411575+0000","last_became_active":"2024-04-04T21:03:39.382089+0000","last_became_peered":"2024-04-04T21:03:39.382089+0000","last_unstale":"2024-04-04T21:39:08.411575+0000","last_undegraded":"2024-04-04T21:39:08.411575+0000","last_fullsized":"2024-04-04T21:39:08.411575+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T01:47:50.490851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2330,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.242110+0000","last_change":"2024-04-04T21:02:24.189161+0000","last_active":"2024-04-04T21:39:09.242110+0000","last_peered":"2024-04-04T21:39:09.242110+0000","last_clean":"2024-04-04T21:39:09.242110+0000","last_became_active":"2024-04-04T21:02:24.159102+0000","last_became_peered":"2024-04-04T21:02:24.159102+0000","last_unstale":"2024-04-04T21:39:09.242110+0000","last_undegraded":"2024-04-04T21:39:09.242110+0000","last_fullsized":"2024-04-04T21:39:09.242110+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:02:24.748876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2296,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.220163+0000","last_change":"2024-04-04T21:03:45.245420+0000","last_active":"2024-04-04T21:39:08.220163+0000","last_peered":"2024-04-04T21:39:08.220163+0000","last_clean":"2024-04-04T21:39:08.220163+0000","last_became_active":"2024-04-04T21:03:45.238093+0000","last_became_peered":"2024-04-04T21:03:45.238093+0000","last_unstale":"2024-04-04T21:39:08.220163+0000","last_undegraded":"2024-04-04T21:39:08.220163+0000","last_fullsized":"2024-04-04T21:39:08.220163+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T06:46:05.953419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2323,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.232461+0000","last_change":"2024-04-04T21:02:24.359272+0000","last_active":"2024-04-04T21:39:09.232461+0000","last_peered":"2024-04-04T21:39:09.232461+0000","last_clean":"2024-04-04T21:39:09.232461+0000","last_became_active":"2024-04-04T21:02:24.352103+0000","last_became_peered":"2024-04-04T21:02:24.352103+0000","last_unstale":"2024-04-04T21:39:09.232461+0000","last_undegraded":"2024-04-04T21:39:09.232461+0000","last_fullsized":"2024-04-04T21:39:09.232461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:36:54.774282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2289,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.382830+0000","last_change":"2024-04-04T21:03:42.104012+0000","last_active":"2024-04-04T21:39:08.382830+0000","last_peered":"2024-04-04T21:39:08.382830+0000","last_clean":"2024-04-04T21:39:08.382830+0000","last_became_active":"2024-04-04T21:03:42.098166+0000","last_became_peered":"2024-04-04T21:03:42.098166+0000","last_unstale":"2024-04-04T21:39:08.382830+0000","last_undegraded":"2024-04-04T21:39:08.382830+0000","last_fullsized":"2024-04-04T21:39:08.382830+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:37:16.291707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2300,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.460408+0000","last_change":"2024-04-04T21:02:24.374143+0000","last_active":"2024-04-04T21:39:08.460408+0000","last_peered":"2024-04-04T21:39:08.460408+0000","last_clean":"2024-04-04T21:39:08.460408+0000","last_became_active":"2024-04-04T21:02:24.359409+0000","last_became_peered":"2024-04-04T21:02:24.359409+0000","last_unstale":"2024-04-04T21:39:08.460408+0000","last_undegraded":"2024-04-04T21:39:08.460408+0000","last_fullsized":"2024-04-04T21:39:08.460408+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:43:28.396677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2305,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.370162+0000","last_change":"2024-04-04T21:03:42.389368+0000","last_active":"2024-04-04T21:39:08.370162+0000","last_peered":"2024-04-04T21:39:08.370162+0000","last_clean":"2024-04-04T21:39:08.370162+0000","last_became_active":"2024-04-04T21:03:42.371847+0000","last_became_peered":"2024-04-04T21:03:42.371847+0000","last_unstale":"2024-04-04T21:39:08.370162+0000","last_undegraded":"2024-04-04T21:39:08.370162+0000","last_fullsized":"2024-04-04T21:39:08.370162+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T06:52:54.899883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2322,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.262941+0000","last_change":"2024-04-04T21:02:24.241805+0000","last_active":"2024-04-04T21:39:09.262941+0000","last_peered":"2024-04-04T21:39:09.262941+0000","last_clean":"2024-04-04T21:39:09.262941+0000","last_became_active":"2024-04-04T21:02:24.236662+0000","last_became_peered":"2024-04-04T21:02:24.236662+0000","last_unstale":"2024-04-04T21:39:09.262941+0000","last_undegraded":"2024-04-04T21:39:09.262941+0000","last_fullsized":"2024-04-04T21:39:09.262941+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:47:33.277018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2306,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.375694+0000","last_change":"2024-04-04T21:03:48.545083+0000","last_active":"2024-04-04T21:39:08.375694+0000","last_peered":"2024-04-04T21:39:08.375694+0000","last_clean":"2024-04-04T21:39:08.375694+0000","last_became_active":"2024-04-04T21:03:48.539598+0000","last_became_peered":"2024-04-04T21:03:48.539598+0000","last_unstale":"2024-04-04T21:39:08.375694+0000","last_undegraded":"2024-04-04T21:39:08.375694+0000","last_fullsized":"2024-04-04T21:39:08.375694+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:16:56.765838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2313,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.389560+0000","last_change":"2024-04-04T21:02:25.438251+0000","last_active":"2024-04-04T21:39:08.389560+0000","last_peered":"2024-04-04T21:39:08.389560+0000","last_clean":"2024-04-04T21:39:08.389560+0000","last_became_active":"2024-04-04T21:02:25.412507+0000","last_became_peered":"2024-04-04T21:02:25.412507+0000","last_unstale":"2024-04-04T21:39:08.389560+0000","last_undegraded":"2024-04-04T21:39:08.389560+0000","last_fullsized":"2024-04-04T21:39:08.389560+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:17:04.580586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2309,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.261110+0000","last_change":"2024-04-04T21:03:56.049841+0000","last_active":"2024-04-04T21:39:09.261110+0000","last_peered":"2024-04-04T21:39:09.261110+0000","last_clean":"2024-04-04T21:39:09.261110+0000","last_became_active":"2024-04-04T21:03:56.035576+0000","last_became_peered":"2024-04-04T21:03:56.035576+0000","last_unstale":"2024-04-04T21:39:09.261110+0000","last_undegraded":"2024-04-04T21:39:09.261110+0000","last_fullsized":"2024-04-04T21:39:09.261110+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:29:26.354814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2321,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.264847+0000","last_change":"2024-04-04T21:02:25.403337+0000","last_active":"2024-04-04T21:39:09.264847+0000","last_peered":"2024-04-04T21:39:09.264847+0000","last_clean":"2024-04-04T21:39:09.264847+0000","last_became_active":"2024-04-04T21:02:25.394276+0000","last_became_peered":"2024-04-04T21:02:25.394276+0000","last_unstale":"2024-04-04T21:39:09.264847+0000","last_undegraded":"2024-04-04T21:39:09.264847+0000","last_fullsized":"2024-04-04T21:39:09.264847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:42:02.653040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2316,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.248209+0000","last_change":"2024-04-04T21:03:55.487132+0000","last_active":"2024-04-04T21:39:09.248209+0000","last_peered":"2024-04-04T21:39:09.248209+0000","last_clean":"2024-04-04T21:39:09.248209+0000","last_became_active":"2024-04-04T21:03:55.479649+0000","last_became_peered":"2024-04-04T21:03:55.479649+0000","last_unstale":"2024-04-04T21:39:09.248209+0000","last_undegraded":"2024-04-04T21:39:09.248209+0000","last_fullsized":"2024-04-04T21:39:09.248209+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:54:34.290339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2333,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.051566+0000","last_change":"2024-04-04T21:03:49.740972+0000","last_active":"2024-04-04T21:39:07.051566+0000","last_peered":"2024-04-04T21:39:07.051566+0000","last_clean":"2024-04-04T21:39:07.051566+0000","last_became_active":"2024-04-04T21:03:49.734675+0000","last_became_peered":"2024-04-04T21:03:49.734675+0000","last_unstale":"2024-04-04T21:39:07.051566+0000","last_undegraded":"2024-04-04T21:39:07.051566+0000","last_fullsized":"2024-04-04T21:39:07.051566+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:02:53.025876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2333,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.063558+0000","last_change":"2024-04-04T21:03:46.411154+0000","last_active":"2024-04-04T21:39:07.063558+0000","last_peered":"2024-04-04T21:39:07.063558+0000","last_clean":"2024-04-04T21:39:07.063558+0000","last_became_active":"2024-04-04T21:03:46.400875+0000","last_became_peered":"2024-04-04T21:03:46.400875+0000","last_unstale":"2024-04-04T21:39:07.063558+0000","last_undegraded":"2024-04-04T21:39:07.063558+0000","last_fullsized":"2024-04-04T21:39:07.063558+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:55:55.301166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2287,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.458495+0000","last_change":"2024-04-04T21:03:47.549919+0000","last_active":"2024-04-04T21:39:08.458495+0000","last_peered":"2024-04-04T21:39:08.458495+0000","last_clean":"2024-04-04T21:39:08.458495+0000","last_became_active":"2024-04-04T21:03:47.514084+0000","last_became_peered":"2024-04-04T21:03:47.514084+0000","last_unstale":"2024-04-04T21:39:08.458495+0000","last_undegraded":"2024-04-04T21:39:08.458495+0000","last_fullsized":"2024-04-04T21:39:08.458495+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T01:39:02.880998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2332,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.048147+0000","last_change":"2024-04-04T21:03:51.865620+0000","last_active":"2024-04-04T21:39:07.048147+0000","last_peered":"2024-04-04T21:39:07.048147+0000","last_clean":"2024-04-04T21:39:07.048147+0000","last_became_active":"2024-04-04T21:03:51.847711+0000","last_became_peered":"2024-04-04T21:03:51.847711+0000","last_unstale":"2024-04-04T21:39:07.048147+0000","last_undegraded":"2024-04-04T21:39:07.048147+0000","last_fullsized":"2024-04-04T21:39:07.048147+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:06:45.768228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2295,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.226407+0000","last_change":"2024-04-04T21:03:40.329642+0000","last_active":"2024-04-04T21:39:08.226407+0000","last_peered":"2024-04-04T21:39:08.226407+0000","last_clean":"2024-04-04T21:39:08.226407+0000","last_became_active":"2024-04-04T21:03:40.322843+0000","last_became_peered":"2024-04-04T21:03:40.322843+0000","last_unstale":"2024-04-04T21:39:08.226407+0000","last_undegraded":"2024-04-04T21:39:08.226407+0000","last_fullsized":"2024-04-04T21:39:08.226407+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T05:04:33.425931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2306,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.379857+0000","last_change":"2024-04-04T21:03:39.843540+0000","last_active":"2024-04-04T21:39:08.379857+0000","last_peered":"2024-04-04T21:39:08.379857+0000","last_clean":"2024-04-04T21:39:08.379857+0000","last_became_active":"2024-04-04T21:03:39.803236+0000","last_became_peered":"2024-04-04T21:03:39.803236+0000","last_unstale":"2024-04-04T21:39:08.379857+0000","last_undegraded":"2024-04-04T21:39:08.379857+0000","last_fullsized":"2024-04-04T21:39:08.379857+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:06:04.364199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2295,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.166370+0000","last_change":"2024-04-04T21:03:40.013569+0000","last_active":"2024-04-04T21:39:08.166370+0000","last_peered":"2024-04-04T21:39:08.166370+0000","last_clean":"2024-04-04T21:39:08.166370+0000","last_became_active":"2024-04-04T21:03:40.004086+0000","last_became_peered":"2024-04-04T21:03:40.004086+0000","last_unstale":"2024-04-04T21:39:08.166370+0000","last_undegraded":"2024-04-04T21:39:08.166370+0000","last_fullsized":"2024-04-04T21:39:08.166370+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T23:37:38.542875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2310,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.229426+0000","last_change":"2024-04-04T21:03:43.781555+0000","last_active":"2024-04-04T21:39:09.229426+0000","last_peered":"2024-04-04T21:39:09.229426+0000","last_clean":"2024-04-04T21:39:09.229426+0000","last_became_active":"2024-04-04T21:03:43.766926+0000","last_became_peered":"2024-04-04T21:03:43.766926+0000","last_unstale":"2024-04-04T21:39:09.229426+0000","last_undegraded":"2024-04-04T21:39:09.229426+0000","last_fullsized":"2024-04-04T21:39:09.229426+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T06:42:24.367113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2283,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.123671+0000","last_change":"2024-04-04T21:03:43.156915+0000","last_active":"2024-04-04T21:39:07.123671+0000","last_peered":"2024-04-04T21:39:07.123671+0000","last_clean":"2024-04-04T21:39:07.123671+0000","last_became_active":"2024-04-04T21:03:43.143344+0000","last_became_peered":"2024-04-04T21:03:43.143344+0000","last_unstale":"2024-04-04T21:39:07.123671+0000","last_undegraded":"2024-04-04T21:39:07.123671+0000","last_fullsized":"2024-04-04T21:39:07.123671+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T22:48:08.738584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2304,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.368340+0000","last_change":"2024-04-04T21:03:42.394120+0000","last_active":"2024-04-04T21:39:08.368340+0000","last_peered":"2024-04-04T21:39:08.368340+0000","last_clean":"2024-04-04T21:39:08.368340+0000","last_became_active":"2024-04-04T21:03:42.379382+0000","last_became_peered":"2024-04-04T21:03:42.379382+0000","last_unstale":"2024-04-04T21:39:08.368340+0000","last_undegraded":"2024-04-04T21:39:08.368340+0000","last_fullsized":"2024-04-04T21:39:08.368340+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:09:54.791797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2288,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.424813+0000","last_change":"2024-04-04T21:03:47.554482+0000","last_active":"2024-04-04T21:39:08.424813+0000","last_peered":"2024-04-04T21:39:08.424813+0000","last_clean":"2024-04-04T21:39:08.424813+0000","last_became_active":"2024-04-04T21:03:47.524251+0000","last_became_peered":"2024-04-04T21:03:47.524251+0000","last_unstale":"2024-04-04T21:39:08.424813+0000","last_undegraded":"2024-04-04T21:39:08.424813+0000","last_fullsized":"2024-04-04T21:39:08.424813+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:11:26.408285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2308,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.252850+0000","last_change":"2024-04-04T21:03:45.814604+0000","last_active":"2024-04-04T21:39:09.252850+0000","last_peered":"2024-04-04T21:39:09.252850+0000","last_clean":"2024-04-04T21:39:09.252850+0000","last_became_active":"2024-04-04T21:03:45.808883+0000","last_became_peered":"2024-04-04T21:03:45.808883+0000","last_unstale":"2024-04-04T21:39:09.252850+0000","last_undegraded":"2024-04-04T21:39:09.252850+0000","last_fullsized":"2024-04-04T21:39:09.252850+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:56:32.831438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899346367,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68324,"kb_used_data":5204,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303516,"statfs":{"total":96636764160,"available":96566800384,"internally_reserved":0,"allocated":5328896,"data_stored":2606125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346367,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72484,"kb_used_data":5204,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299356,"statfs":{"total":96636764160,"available":96562540544,"internally_reserved":0,"allocated":5328896,"data_stored":2606125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346367,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72856,"kb_used_data":5580,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298984,"statfs":{"total":96636764160,"available":96562159616,"internally_reserved":0,"allocated":5713920,"data_stored":2996281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411777,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72484,"kb_used_data":5208,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299356,"statfs":{"total":96636764160,"available":96562540544,"internally_reserved":0,"allocated":5332992,"data_stored":2606130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509889,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72468,"kb_used_data":5188,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299372,"statfs":{"total":96636764160,"available":96562556928,"internally_reserved":0,"allocated":5312512,"data_stored":2598441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509884,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68300,"kb_used_data":5176,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303540,"statfs":{"total":96636764160,"available":96566824960,"internally_reserved":0,"allocated":5300224,"data_stored":2590993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542591,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72500,"kb_used_data":5220,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299340,"statfs":{"total":96636764160,"available":96562524160,"internally_reserved":0,"allocated":5345280,"data_stored":2612837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575293,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68712,"kb_used_data":5596,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303128,"statfs":{"total":96636764160,"available":96566403072,"internally_reserved":0,"allocated":5730304,"data_stored":3003965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-04T21:39:11.543 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-04T21:39:11.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.646+0000 7fc848fc5640 10 monclient: get_monmap_and_config 2024-04-04T21:39:11.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.646+0000 7fc848fc5640 10 monclient: build_initial_monmap 2024-04-04T21:39:11.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.646+0000 7fc848fc5640 10 monclient: monmap: 2024-04-04T21:39:11.647 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:11.647 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:11.647 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:11.648025+0000 2024-04-04T21:39:11.647 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:11.648025+0000 2024-04-04T21:39:11.647 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:11.647 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:11.647 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:11.648 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:11.648 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:11.648 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:11.648 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:11.648 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:11.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.647+0000 7fc848fc5640 1 Processor -- start 2024-04-04T21:39:11.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.647+0000 7fc848fc5640 1 -- start start 2024-04-04T21:39:11.649 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.647+0000 7fc848fc5640 10 monclient: init 2024-04-04T21:39:11.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.648+0000 7fc848fc5640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:11.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.648+0000 7fc848fc5640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-04-04T21:39:11.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.648+0000 7fc848fc5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8440a9b60 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:11.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.648+0000 7fc848fc5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc8441426f0 0x7fc8440b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:11.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.648+0000 7fc848fc5640 10 monclient(hunting): picked mon.noname-c con 0x7fc8441426f0 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:11.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.648+0000 7fc848fc5640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc84413f810 0x7fc84413fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:11.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.648+0000 7fc848fc5640 10 monclient(hunting): picked mon.noname-b con 0x7fc84413f810 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:11.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.648+0000 7fc848fc5640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8440b2e00 con 0x7fc8440a9b60 2024-04-04T21:39:11.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.648+0000 7fc848fc5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:11.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.648+0000 7fc848fc5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8440a9f40 con 0x7fc8441426f0 2024-04-04T21:39:11.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.648+0000 7fc848fc5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:11.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.648+0000 7fc848fc5640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc844142c10 con 0x7fc84413f810 2024-04-04T21:39:11.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.648+0000 7fc848fc5640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:11.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.648+0000 7fc848fc5640 10 monclient(hunting): authenticate will time out at 3600.915563s 2024-04-04T21:39:11.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.650+0000 7fc842d9d640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc84413f810 0x7fc84413fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:11.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.651+0000 7fc84259c640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fc8440a9b60 0x7fc8441430a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:55134/0 (socket says 172.21.15.106:55134) 2024-04-04T21:39:11.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.651+0000 7fc84259c640 1 -- 172.21.15.106:0/3943912593 learned_addr learned my addr 172.21.15.106:0/3943912593 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:39:11.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.652+0000 7fc842d9d640 10 monclient(hunting): get_auth_request con 0x7fc84413f810 auth_method 0 2024-04-04T21:39:11.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.652+0000 7fc842d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:11.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.652+0000 7fc842d9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:11.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.652+0000 7fc842d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:11.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.654+0000 7fc842d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:11.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.654+0000 7fc842d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:11.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.654+0000 7fc842d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:11.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.656+0000 7fc842d9d640 10 monclient(hunting): handle_auth_done global_id 17627 payload 274 2024-04-04T21:39:11.657 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.656+0000 7fc842d9d640 1 -- 172.21.15.106:0/3943912593 >> v2:172.21.15.203:3300/0 conn(0x7fc8441426f0 msgr2=0x7fc8440b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:11.657 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.656+0000 7fc842d9d640 1 --2- 172.21.15.106:0/3943912593 >> v2:172.21.15.203:3300/0 conn(0x7fc8441426f0 0x7fc8440b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.657 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.656+0000 7fc842d9d640 1 -- 172.21.15.106:0/3943912593 >> v1:172.21.15.106:6789/0 conn(0x7fc8440a9b60 legacy=0x7fc8441430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:11.657 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.656+0000 7fc842d9d640 10 monclient: _finish_hunting 0 2024-04-04T21:39:11.657 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.656+0000 7fc842d9d640 1 monclient: found mon.noname-b 2024-04-04T21:39:11.657 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.656+0000 7fc842d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:11.657 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.656+0000 7fc842d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:39:11.657 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.656+0000 7fc842d9d640 1 -- 172.21.15.106:0/3943912593 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8440b8c90 con 0x7fc84413f810 2024-04-04T21:39:11.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.663+0000 7fc842d9d640 1 --2- 172.21.15.106:0/3943912593 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc84413f810 0x7fc84413fcc0 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009d90 tx=0x7fc83c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eca5cd5dc007fa67 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:11.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc841d9b640 1 -- 172.21.15.106:0/3943912593 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c039070 con 0x7fc84413f810 2024-04-04T21:39:11.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc841d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:11.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc841d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:11.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc841d9b640 10 monclient: dump: 2024-04-04T21:39:11.680 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:11.680 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:11.681 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.681 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.681 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:11.681 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:11.681 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:11.681 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:11.681 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:11.681 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:11.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc841d9b640 10 monclient: _finish_auth 0 2024-04-04T21:39:11.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc841d9b640 10 monclient: _check_auth_tickets 2024-04-04T21:39:11.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc841d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:11.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc841d9b640 1 -- 172.21.15.106:0/3943912593 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83c02cbc0 con 0x7fc84413f810 2024-04-04T21:39:11.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc848fc5640 5 monclient: authenticate success, global_id 17627 2024-04-04T21:39:11.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc848fc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:39:11.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc841d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:11.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc841d9b640 1 -- 172.21.15.106:0/3943912593 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c034440 con 0x7fc84413f810 2024-04-04T21:39:11.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc841d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:11.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc841d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:39:11.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc841d9b640 10 monclient: dump: 2024-04-04T21:39:11.682 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:11.682 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:11.682 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.682 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.682 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:11.682 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:11.682 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:11.682 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:11.682 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:11.683 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:11.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc848fc5640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:11.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.680+0000 7fc848fc5640 4 set_mon_vals no callback set 2024-04-04T21:39:11.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.681+0000 7fc848fc5640 10 monclient: shutdown 2024-04-04T21:39:11.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.681+0000 7fc848fc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:11.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.681+0000 7fc848fc5640 1 -- 172.21.15.106:0/3943912593 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc84413f810 msgr2=0x7fc84413fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:11.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.681+0000 7fc848fc5640 1 --2- 172.21.15.106:0/3943912593 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc84413f810 0x7fc84413fcc0 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009d90 tx=0x7fc83c02be10 comp rx=0 tx=0).stop 2024-04-04T21:39:11.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.681+0000 7fc848fc5640 1 -- 172.21.15.106:0/3943912593 shutdown_connections 2024-04-04T21:39:11.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.681+0000 7fc848fc5640 1 --2- 172.21.15.106:0/3943912593 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc84413f810 0x7fc84413fcc0 unknown :-1 s=CLOSED pgs=3265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.681+0000 7fc848fc5640 1 --2- 172.21.15.106:0/3943912593 >> v2:172.21.15.203:3300/0 conn(0x7fc8441426f0 0x7fc8440b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.681+0000 7fc848fc5640 1 -- 172.21.15.106:0/3943912593 >> 172.21.15.106:0/3943912593 conn(0x7fc8440b7ec0 msgr2=0x7fc8440b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:11.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.681+0000 7fc848fc5640 1 -- 172.21.15.106:0/3943912593 shutdown_connections 2024-04-04T21:39:11.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.681+0000 7fc848fc5640 1 -- 172.21.15.106:0/3943912593 wait complete. 2024-04-04T21:39:11.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 10 monclient: build_initial_monmap 2024-04-04T21:39:11.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 10 monclient: monmap: 2024-04-04T21:39:11.686 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:11.686 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:11.686 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:11.686 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:11.686 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:11.686 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:11.686 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:11.686 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:11.686 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:11.686 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:11.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 1 librados: starting msgr at 2024-04-04T21:39:11.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 1 librados: starting objecter 2024-04-04T21:39:11.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 1 Processor -- start 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 1 -- start start 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 1 librados: setting wanted keys 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 1 librados: calling monclient init 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 10 monclient: init 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8440a9b60 0x7fc84413ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 10 monclient(hunting): picked mon.noname-b con 0x7fc8440a9b60 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc84413f810 0x7fc84413b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 10 monclient(hunting): picked mon.noname-a con 0x7fc84413f810 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8441d7d80 0x7fc8441d8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 10 monclient(hunting): picked mon.noname-c con 0x7fc8441d7d80 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8440b2530 con 0x7fc84413f810 2024-04-04T21:39:11.687 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:11.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8440b23b0 con 0x7fc8440a9b60 2024-04-04T21:39:11.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:11.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc84414d000 con 0x7fc8441d7d80 2024-04-04T21:39:11.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:11.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.682+0000 7fc848fc5640 10 monclient(hunting): authenticate will time out at 3600.950174s 2024-04-04T21:39:11.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.686+0000 7fc83bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc84413f810 0x7fc84413b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:11.688 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.686+0000 7fc83bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc84413f810 0x7fc84413b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:60892/0 (socket says 172.21.15.106:60892) 2024-04-04T21:39:11.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.686+0000 7fc83bfff640 1 -- 172.21.15.106:0/1069385386 learned_addr learned my addr 172.21.15.106:0/1069385386 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:11.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.686+0000 7fc83bfff640 10 monclient(hunting): get_auth_request con 0x7fc84413f810 auth_method 0 2024-04-04T21:39:11.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.686+0000 7fc83bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:11.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.686+0000 7fc83bfff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:11.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.686+0000 7fc83bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:11.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.687+0000 7fc842d9d640 1 --2- 172.21.15.106:0/1069385386 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8441d7d80 0x7fc8441d8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:11.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.691+0000 7fc842d9d640 10 monclient(hunting): get_auth_request con 0x7fc8441d7d80 auth_method 0 2024-04-04T21:39:11.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.691+0000 7fc842d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:11.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.691+0000 7fc842d9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:11.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.691+0000 7fc842d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:11.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.691+0000 7fc83bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:11.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.691+0000 7fc83bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:11.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.691+0000 7fc83bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:11.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.692+0000 7fc842d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:11.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.692+0000 7fc842d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:11.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.692+0000 7fc842d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:11.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.695+0000 7fc83bfff640 10 monclient(hunting): handle_auth_done global_id 15090 payload 824 2024-04-04T21:39:11.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.695+0000 7fc83bfff640 1 -- 172.21.15.106:0/1069385386 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8441d7d80 msgr2=0x7fc8441d8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:11.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.695+0000 7fc83bfff640 1 --2- 172.21.15.106:0/1069385386 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8441d7d80 0x7fc8441d8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.695+0000 7fc83bfff640 1 -- 172.21.15.106:0/1069385386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8440a9b60 msgr2=0x7fc84413ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:11.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.695+0000 7fc83bfff640 1 --2- 172.21.15.106:0/1069385386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8440a9b60 0x7fc84413ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:11.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.695+0000 7fc83bfff640 10 monclient: _finish_hunting 0 2024-04-04T21:39:11.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.695+0000 7fc83bfff640 1 monclient: found mon.noname-a 2024-04-04T21:39:11.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.695+0000 7fc83bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:11.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.695+0000 7fc83bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:39:11.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.695+0000 7fc83bfff640 1 -- 172.21.15.106:0/1069385386 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8440ac680 con 0x7fc84413f810 2024-04-04T21:39:11.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.707+0000 7fc83bfff640 1 --2- 172.21.15.106:0/1069385386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc84413f810 0x7fc84413b680 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00b4d0 tx=0x7fc82c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:11.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.707+0000 7fc83affd640 10 client.?.objecter ms_handle_connect 0x7fc84413f810 2024-04-04T21:39:11.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.707+0000 7fc83affd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:11.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc83affd640 1 -- 172.21.15.106:0/1069385386 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c014070 con 0x7fc84413f810 2024-04-04T21:39:11.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc83affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:11.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc83affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:39:11.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc83affd640 10 monclient: dump: 2024-04-04T21:39:11.722 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:11.722 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:11.722 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.722 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.722 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:11.722 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:11.722 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:11.723 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:11.723 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:11.723 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:11.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc83affd640 10 monclient: _finish_auth 0 2024-04-04T21:39:11.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc83affd640 10 monclient: _check_auth_tickets 2024-04-04T21:39:11.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc83affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:11.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc83affd640 1 -- 172.21.15.106:0/1069385386 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82c010870 con 0x7fc84413f810 2024-04-04T21:39:11.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc83affd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:11.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc83affd640 1 -- 172.21.15.106:0/1069385386 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c010c60 con 0x7fc84413f810 2024-04-04T21:39:11.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc848fc5640 5 monclient: authenticate success, global_id 15090 2024-04-04T21:39:11.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc848fc5640 10 monclient: _renew_subs 2024-04-04T21:39:11.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc848fc5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:39:11.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc84159a640 4 set_mon_vals no callback set 2024-04-04T21:39:11.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc848fc5640 1 -- 172.21.15.106:0/1069385386 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8441d8770 con 0x7fc84413f810 2024-04-04T21:39:11.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc83affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:11.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc83affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:39:11.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc83affd640 10 monclient: dump: 2024-04-04T21:39:11.724 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:11.724 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:11.724 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.724 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:11.724 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:11.724 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:11.724 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:11.724 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:11.724 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:11.724 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:11.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc848fc5640 10 client.15090.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:11.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc848fc5640 10 monclient: _renew_subs 2024-04-04T21:39:11.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc848fc5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:39:11.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc848fc5640 1 -- 172.21.15.106:0/1069385386 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84413bbc0 con 0x7fc84413f810 2024-04-04T21:39:11.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.721+0000 7fc848fc5640 1 librados: init done 2024-04-04T21:39:11.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.722+0000 7fc848fc5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:11.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.722+0000 7fc848fc5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:11.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.722+0000 7fc848fc5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:39:11.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.722+0000 7fc848fc5640 1 -- 172.21.15.106:0/1069385386 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8440ac480 con 0x7fc84413f810 2024-04-04T21:39:11.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.998+0000 7fc83affd640 1 -- 172.21.15.106:0/1069385386 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc82c017070 con 0x7fc84413f810 2024-04-04T21:39:11.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.999+0000 7fc83affd640 1 --2- 172.21.15.106:0/1069385386 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc82406ceb0 0x7fc82406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:11.999 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.999+0000 7fc84259c640 1 --2- 172.21.15.106:0/1069385386 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc82406ceb0 0x7fc82406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:12.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.999+0000 7fc84259c640 10 monclient: get_auth_request con 0x7fc82406ceb0 auth_method 0 2024-04-04T21:39:12.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.999+0000 7fc84259c640 1 --2- 172.21.15.106:0/1069385386 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc82406ceb0 0x7fc82406f370 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc834000f60 tx=0x7fc834007a80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:12.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:11.999+0000 7fc83affd640 10 client.15090.objecter ms_handle_connect 0x7fc82406ceb0 2024-04-04T21:39:12.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.012+0000 7fc83affd640 1 -- 172.21.15.106:0/1069385386 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7fc82c056c10 con 0x7fc84413f810 2024-04-04T21:39:12.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.012+0000 7fc83affd640 10 client.15090.objecter ms_dispatch 0x7fc844000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:12.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.012+0000 7fc83affd640 3 client.15090.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:12.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.012+0000 7fc83affd640 3 client.15090.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:12.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.012+0000 7fc83affd640 20 client.15090.objecter dump_active .. 0 homeless 2024-04-04T21:39:12.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.012+0000 7fc83affd640 1 -- 172.21.15.106:0/1069385386 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc82c058b70 con 0x7fc84413f810 2024-04-04T21:39:12.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.012+0000 7fc83affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:12.013 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.012+0000 7fc83affd640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:12.200 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.200+0000 7fc848fc5640 1 -- 172.21.15.106:0/1069385386 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc84404f050 con 0x7fc82406ceb0 2024-04-04T21:39:12.203 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.203+0000 7fc83affd640 1 -- 172.21.15.106:0/1069385386 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+116264 (secure 0 0 0) 0x7fc84404f050 con 0x7fc82406ceb0 2024-04-04T21:39:12.204 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:39:12.205 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-04-04T21:39:12.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 10 librados: watch_flush enter 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 10 librados: watch_flush exit 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 20 client.15090.objecter shutdown clearing up homeless session... 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 10 client.15090.objecter successfully canceled tick 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 1 -- 172.21.15.106:0/1069385386 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc82406ceb0 msgr2=0x7fc82406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 1 --2- 172.21.15.106:0/1069385386 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc82406ceb0 0x7fc82406f370 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc834000f60 tx=0x7fc834007a80 comp rx=0 tx=0).stop 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 10 monclient: shutdown 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 1 -- 172.21.15.106:0/1069385386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc84413f810 msgr2=0x7fc84413b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 1 --2- 172.21.15.106:0/1069385386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc84413f810 0x7fc84413b680 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00b4d0 tx=0x7fc82c00b990 comp rx=0 tx=0).stop 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 1 -- 172.21.15.106:0/1069385386 shutdown_connections 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 1 --2- 172.21.15.106:0/1069385386 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fc82406ceb0 0x7fc82406f370 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 1 --2- 172.21.15.106:0/1069385386 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8441d7d80 0x7fc8441d8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 1 --2- 172.21.15.106:0/1069385386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc84413f810 0x7fc84413b680 unknown :-1 s=CLOSED pgs=3157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 1 --2- 172.21.15.106:0/1069385386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8440a9b60 0x7fc84413ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.205+0000 7fc848fc5640 1 -- 172.21.15.106:0/1069385386 >> 172.21.15.106:0/1069385386 conn(0x7fc8440b7ec0 msgr2=0x7fc8440b50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:12.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.206+0000 7fc848fc5640 1 -- 172.21.15.106:0/1069385386 shutdown_connections 2024-04-04T21:39:12.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.206+0000 7fc848fc5640 1 -- 172.21.15.106:0/1069385386 wait complete. 2024-04-04T21:39:12.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.206+0000 7fc848fc5640 1 librados: shutdown 2024-04-04T21:39:12.217 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":2312,"stamp":"2024-04-04T21:39:11.059877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":87,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":568128,"kb_used_data":42376,"kb_used_omap":12,"kb_used_meta":525619,"kb_avail":754406592,"statfs":{"total":773094113280,"available":772512350208,"internally_reserved":0,"allocated":43393024,"data_stored":21620897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":538234417},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":158,"apply_latency_ms":158,"commit_latency_ns":158000000,"apply_latency_ns":158000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.996579"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2332,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.039530+0000","last_change":"2024-04-04T21:03:54.942652+0000","last_active":"2024-04-04T21:39:07.039530+0000","last_peered":"2024-04-04T21:39:07.039530+0000","last_clean":"2024-04-04T21:39:07.039530+0000","last_became_active":"2024-04-04T21:03:54.933965+0000","last_became_peered":"2024-04-04T21:03:54.933965+0000","last_unstale":"2024-04-04T21:39:07.039530+0000","last_undegraded":"2024-04-04T21:39:07.039530+0000","last_fullsized":"2024-04-04T21:39:07.039530+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:47:55.727002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2305,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.360894+0000","last_change":"2024-04-04T21:03:44.565986+0000","last_active":"2024-04-04T21:39:08.360894+0000","last_peered":"2024-04-04T21:39:08.360894+0000","last_clean":"2024-04-04T21:39:08.360894+0000","last_became_active":"2024-04-04T21:03:44.551107+0000","last_became_peered":"2024-04-04T21:03:44.551107+0000","last_unstale":"2024-04-04T21:39:08.360894+0000","last_undegraded":"2024-04-04T21:39:08.360894+0000","last_fullsized":"2024-04-04T21:39:08.360894+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:51:23.943936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2301,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.405327+0000","last_change":"2024-04-04T21:03:52.793808+0000","last_active":"2024-04-04T21:39:08.405327+0000","last_peered":"2024-04-04T21:39:08.405327+0000","last_clean":"2024-04-04T21:39:08.405327+0000","last_became_active":"2024-04-04T21:03:52.772128+0000","last_became_peered":"2024-04-04T21:03:52.772128+0000","last_unstale":"2024-04-04T21:39:08.405327+0000","last_undegraded":"2024-04-04T21:39:08.405327+0000","last_fullsized":"2024-04-04T21:39:08.405327+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:44:28.885034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2287,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.407788+0000","last_change":"2024-04-04T21:03:41.361551+0000","last_active":"2024-04-04T21:39:08.407788+0000","last_peered":"2024-04-04T21:39:08.407788+0000","last_clean":"2024-04-04T21:39:08.407788+0000","last_became_active":"2024-04-04T21:03:41.346161+0000","last_became_peered":"2024-04-04T21:03:41.346161+0000","last_unstale":"2024-04-04T21:39:08.407788+0000","last_undegraded":"2024-04-04T21:39:08.407788+0000","last_fullsized":"2024-04-04T21:39:08.407788+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:14:45.723197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2289,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.440013+0000","last_change":"2024-04-04T21:03:42.130175+0000","last_active":"2024-04-04T21:39:08.440013+0000","last_peered":"2024-04-04T21:39:08.440013+0000","last_clean":"2024-04-04T21:39:08.440013+0000","last_became_active":"2024-04-04T21:03:42.122180+0000","last_became_peered":"2024-04-04T21:03:42.122180+0000","last_unstale":"2024-04-04T21:39:08.440013+0000","last_undegraded":"2024-04-04T21:39:08.440013+0000","last_fullsized":"2024-04-04T21:39:08.440013+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:01:39.698809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2318,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.250973+0000","last_change":"2024-04-04T21:03:42.949631+0000","last_active":"2024-04-04T21:39:09.250973+0000","last_peered":"2024-04-04T21:39:09.250973+0000","last_clean":"2024-04-04T21:39:09.250973+0000","last_became_active":"2024-04-04T21:03:42.940728+0000","last_became_peered":"2024-04-04T21:03:42.940728+0000","last_unstale":"2024-04-04T21:39:09.250973+0000","last_undegraded":"2024-04-04T21:39:09.250973+0000","last_fullsized":"2024-04-04T21:39:09.250973+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:24:58.260325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2306,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.372867+0000","last_change":"2024-04-04T21:03:43.791130+0000","last_active":"2024-04-04T21:39:08.372867+0000","last_peered":"2024-04-04T21:39:08.372867+0000","last_clean":"2024-04-04T21:39:08.372867+0000","last_became_active":"2024-04-04T21:03:43.596620+0000","last_became_peered":"2024-04-04T21:03:43.596620+0000","last_unstale":"2024-04-04T21:39:08.372867+0000","last_undegraded":"2024-04-04T21:39:08.372867+0000","last_fullsized":"2024-04-04T21:39:08.372867+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T21:23:20.072459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2307,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.377536+0000","last_change":"2024-04-04T21:03:52.362151+0000","last_active":"2024-04-04T21:39:08.377536+0000","last_peered":"2024-04-04T21:39:08.377536+0000","last_clean":"2024-04-04T21:39:08.377536+0000","last_became_active":"2024-04-04T21:03:52.351392+0000","last_became_peered":"2024-04-04T21:03:52.351392+0000","last_unstale":"2024-04-04T21:39:08.377536+0000","last_undegraded":"2024-04-04T21:39:08.377536+0000","last_fullsized":"2024-04-04T21:39:08.377536+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T01:47:25.888369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"39'7","reported_seq":2332,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.239999+0000","last_change":"2024-04-04T21:03:50.565469+0000","last_active":"2024-04-04T21:39:09.239999+0000","last_peered":"2024-04-04T21:39:09.239999+0000","last_clean":"2024-04-04T21:39:09.239999+0000","last_became_active":"2024-04-04T21:03:50.287197+0000","last_became_peered":"2024-04-04T21:03:50.287197+0000","last_unstale":"2024-04-04T21:39:09.239999+0000","last_undegraded":"2024-04-04T21:39:09.239999+0000","last_fullsized":"2024-04-04T21:39:09.239999+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T04:15:05.649671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2290,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.366167+0000","last_change":"2024-04-04T21:03:44.924431+0000","last_active":"2024-04-04T21:39:08.366167+0000","last_peered":"2024-04-04T21:39:08.366167+0000","last_clean":"2024-04-04T21:39:08.366167+0000","last_became_active":"2024-04-04T21:03:44.913048+0000","last_became_peered":"2024-04-04T21:03:44.913048+0000","last_unstale":"2024-04-04T21:39:08.366167+0000","last_undegraded":"2024-04-04T21:39:08.366167+0000","last_fullsized":"2024-04-04T21:39:08.366167+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T22:52:04.454928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2286,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.112318+0000","last_change":"2024-04-04T21:03:44.496009+0000","last_active":"2024-04-04T21:39:07.112318+0000","last_peered":"2024-04-04T21:39:07.112318+0000","last_clean":"2024-04-04T21:39:07.112318+0000","last_became_active":"2024-04-04T21:03:44.451112+0000","last_became_peered":"2024-04-04T21:03:44.451112+0000","last_unstale":"2024-04-04T21:39:07.112318+0000","last_undegraded":"2024-04-04T21:39:07.112318+0000","last_fullsized":"2024-04-04T21:39:07.112318+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:12:30.730952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2334,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.024282+0000","last_change":"2024-04-04T21:03:44.379999+0000","last_active":"2024-04-04T21:39:07.024282+0000","last_peered":"2024-04-04T21:39:07.024282+0000","last_clean":"2024-04-04T21:39:07.024282+0000","last_became_active":"2024-04-04T21:03:44.366452+0000","last_became_peered":"2024-04-04T21:03:44.366452+0000","last_unstale":"2024-04-04T21:39:07.024282+0000","last_undegraded":"2024-04-04T21:39:07.024282+0000","last_fullsized":"2024-04-04T21:39:07.024282+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:21:39.463272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":2318,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.168312+0000","last_change":"2024-04-04T21:02:20.509950+0000","last_active":"2024-04-04T21:39:08.168312+0000","last_peered":"2024-04-04T21:39:08.168312+0000","last_clean":"2024-04-04T21:39:08.168312+0000","last_became_active":"2024-04-04T21:02:19.876220+0000","last_became_peered":"2024-04-04T21:02:19.876220+0000","last_unstale":"2024-04-04T21:39:08.168312+0000","last_undegraded":"2024-04-04T21:39:08.168312+0000","last_fullsized":"2024-04-04T21:39:08.168312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:05.463818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:05.463818+0000","last_clean_scrub_stamp":"2024-04-04T21:02:05.463818+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:18:20.734360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2323,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.257109+0000","last_change":"2024-04-04T21:02:24.476744+0000","last_active":"2024-04-04T21:39:09.257109+0000","last_peered":"2024-04-04T21:39:09.257109+0000","last_clean":"2024-04-04T21:39:09.257109+0000","last_became_active":"2024-04-04T21:02:24.466203+0000","last_became_peered":"2024-04-04T21:02:24.466203+0000","last_unstale":"2024-04-04T21:39:09.257109+0000","last_undegraded":"2024-04-04T21:39:09.257109+0000","last_fullsized":"2024-04-04T21:39:09.257109+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T21:40:50.021063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2289,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.369121+0000","last_change":"2024-04-04T21:03:40.772628+0000","last_active":"2024-04-04T21:39:08.369121+0000","last_peered":"2024-04-04T21:39:08.369121+0000","last_clean":"2024-04-04T21:39:08.369121+0000","last_became_active":"2024-04-04T21:03:40.758544+0000","last_became_peered":"2024-04-04T21:03:40.758544+0000","last_unstale":"2024-04-04T21:39:08.369121+0000","last_undegraded":"2024-04-04T21:39:08.369121+0000","last_fullsized":"2024-04-04T21:39:08.369121+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:22:11.238559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2324,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.235421+0000","last_change":"2024-04-04T21:02:24.174900+0000","last_active":"2024-04-04T21:39:09.235421+0000","last_peered":"2024-04-04T21:39:09.235421+0000","last_clean":"2024-04-04T21:39:09.235421+0000","last_became_active":"2024-04-04T21:02:24.146676+0000","last_became_peered":"2024-04-04T21:02:24.146676+0000","last_unstale":"2024-04-04T21:39:09.235421+0000","last_undegraded":"2024-04-04T21:39:09.235421+0000","last_fullsized":"2024-04-04T21:39:09.235421+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:47:28.827956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2287,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.411575+0000","last_change":"2024-04-04T21:03:39.388042+0000","last_active":"2024-04-04T21:39:08.411575+0000","last_peered":"2024-04-04T21:39:08.411575+0000","last_clean":"2024-04-04T21:39:08.411575+0000","last_became_active":"2024-04-04T21:03:39.382089+0000","last_became_peered":"2024-04-04T21:03:39.382089+0000","last_unstale":"2024-04-04T21:39:08.411575+0000","last_undegraded":"2024-04-04T21:39:08.411575+0000","last_fullsized":"2024-04-04T21:39:08.411575+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T01:47:50.490851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2330,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.242110+0000","last_change":"2024-04-04T21:02:24.189161+0000","last_active":"2024-04-04T21:39:09.242110+0000","last_peered":"2024-04-04T21:39:09.242110+0000","last_clean":"2024-04-04T21:39:09.242110+0000","last_became_active":"2024-04-04T21:02:24.159102+0000","last_became_peered":"2024-04-04T21:02:24.159102+0000","last_unstale":"2024-04-04T21:39:09.242110+0000","last_undegraded":"2024-04-04T21:39:09.242110+0000","last_fullsized":"2024-04-04T21:39:09.242110+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:02:24.748876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2296,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.220163+0000","last_change":"2024-04-04T21:03:45.245420+0000","last_active":"2024-04-04T21:39:08.220163+0000","last_peered":"2024-04-04T21:39:08.220163+0000","last_clean":"2024-04-04T21:39:08.220163+0000","last_became_active":"2024-04-04T21:03:45.238093+0000","last_became_peered":"2024-04-04T21:03:45.238093+0000","last_unstale":"2024-04-04T21:39:08.220163+0000","last_undegraded":"2024-04-04T21:39:08.220163+0000","last_fullsized":"2024-04-04T21:39:08.220163+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T06:46:05.953419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2323,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.232461+0000","last_change":"2024-04-04T21:02:24.359272+0000","last_active":"2024-04-04T21:39:09.232461+0000","last_peered":"2024-04-04T21:39:09.232461+0000","last_clean":"2024-04-04T21:39:09.232461+0000","last_became_active":"2024-04-04T21:02:24.352103+0000","last_became_peered":"2024-04-04T21:02:24.352103+0000","last_unstale":"2024-04-04T21:39:09.232461+0000","last_undegraded":"2024-04-04T21:39:09.232461+0000","last_fullsized":"2024-04-04T21:39:09.232461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:36:54.774282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2289,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.382830+0000","last_change":"2024-04-04T21:03:42.104012+0000","last_active":"2024-04-04T21:39:08.382830+0000","last_peered":"2024-04-04T21:39:08.382830+0000","last_clean":"2024-04-04T21:39:08.382830+0000","last_became_active":"2024-04-04T21:03:42.098166+0000","last_became_peered":"2024-04-04T21:03:42.098166+0000","last_unstale":"2024-04-04T21:39:08.382830+0000","last_undegraded":"2024-04-04T21:39:08.382830+0000","last_fullsized":"2024-04-04T21:39:08.382830+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:37:16.291707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2300,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.460408+0000","last_change":"2024-04-04T21:02:24.374143+0000","last_active":"2024-04-04T21:39:08.460408+0000","last_peered":"2024-04-04T21:39:08.460408+0000","last_clean":"2024-04-04T21:39:08.460408+0000","last_became_active":"2024-04-04T21:02:24.359409+0000","last_became_peered":"2024-04-04T21:02:24.359409+0000","last_unstale":"2024-04-04T21:39:08.460408+0000","last_undegraded":"2024-04-04T21:39:08.460408+0000","last_fullsized":"2024-04-04T21:39:08.460408+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:43:28.396677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2305,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.370162+0000","last_change":"2024-04-04T21:03:42.389368+0000","last_active":"2024-04-04T21:39:08.370162+0000","last_peered":"2024-04-04T21:39:08.370162+0000","last_clean":"2024-04-04T21:39:08.370162+0000","last_became_active":"2024-04-04T21:03:42.371847+0000","last_became_peered":"2024-04-04T21:03:42.371847+0000","last_unstale":"2024-04-04T21:39:08.370162+0000","last_undegraded":"2024-04-04T21:39:08.370162+0000","last_fullsized":"2024-04-04T21:39:08.370162+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T06:52:54.899883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2322,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.262941+0000","last_change":"2024-04-04T21:02:24.241805+0000","last_active":"2024-04-04T21:39:09.262941+0000","last_peered":"2024-04-04T21:39:09.262941+0000","last_clean":"2024-04-04T21:39:09.262941+0000","last_became_active":"2024-04-04T21:02:24.236662+0000","last_became_peered":"2024-04-04T21:02:24.236662+0000","last_unstale":"2024-04-04T21:39:09.262941+0000","last_undegraded":"2024-04-04T21:39:09.262941+0000","last_fullsized":"2024-04-04T21:39:09.262941+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:47:33.277018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2306,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.375694+0000","last_change":"2024-04-04T21:03:48.545083+0000","last_active":"2024-04-04T21:39:08.375694+0000","last_peered":"2024-04-04T21:39:08.375694+0000","last_clean":"2024-04-04T21:39:08.375694+0000","last_became_active":"2024-04-04T21:03:48.539598+0000","last_became_peered":"2024-04-04T21:03:48.539598+0000","last_unstale":"2024-04-04T21:39:08.375694+0000","last_undegraded":"2024-04-04T21:39:08.375694+0000","last_fullsized":"2024-04-04T21:39:08.375694+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:16:56.765838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2313,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.389560+0000","last_change":"2024-04-04T21:02:25.438251+0000","last_active":"2024-04-04T21:39:08.389560+0000","last_peered":"2024-04-04T21:39:08.389560+0000","last_clean":"2024-04-04T21:39:08.389560+0000","last_became_active":"2024-04-04T21:02:25.412507+0000","last_became_peered":"2024-04-04T21:02:25.412507+0000","last_unstale":"2024-04-04T21:39:08.389560+0000","last_undegraded":"2024-04-04T21:39:08.389560+0000","last_fullsized":"2024-04-04T21:39:08.389560+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:17:04.580586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2309,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.261110+0000","last_change":"2024-04-04T21:03:56.049841+0000","last_active":"2024-04-04T21:39:09.261110+0000","last_peered":"2024-04-04T21:39:09.261110+0000","last_clean":"2024-04-04T21:39:09.261110+0000","last_became_active":"2024-04-04T21:03:56.035576+0000","last_became_peered":"2024-04-04T21:03:56.035576+0000","last_unstale":"2024-04-04T21:39:09.261110+0000","last_undegraded":"2024-04-04T21:39:09.261110+0000","last_fullsized":"2024-04-04T21:39:09.261110+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:29:26.354814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2321,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.264847+0000","last_change":"2024-04-04T21:02:25.403337+0000","last_active":"2024-04-04T21:39:09.264847+0000","last_peered":"2024-04-04T21:39:09.264847+0000","last_clean":"2024-04-04T21:39:09.264847+0000","last_became_active":"2024-04-04T21:02:25.394276+0000","last_became_peered":"2024-04-04T21:02:25.394276+0000","last_unstale":"2024-04-04T21:39:09.264847+0000","last_undegraded":"2024-04-04T21:39:09.264847+0000","last_fullsized":"2024-04-04T21:39:09.264847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:42:02.653040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2316,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.248209+0000","last_change":"2024-04-04T21:03:55.487132+0000","last_active":"2024-04-04T21:39:09.248209+0000","last_peered":"2024-04-04T21:39:09.248209+0000","last_clean":"2024-04-04T21:39:09.248209+0000","last_became_active":"2024-04-04T21:03:55.479649+0000","last_became_peered":"2024-04-04T21:03:55.479649+0000","last_unstale":"2024-04-04T21:39:09.248209+0000","last_undegraded":"2024-04-04T21:39:09.248209+0000","last_fullsized":"2024-04-04T21:39:09.248209+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:54:34.290339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2333,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.051566+0000","last_change":"2024-04-04T21:03:49.740972+0000","last_active":"2024-04-04T21:39:07.051566+0000","last_peered":"2024-04-04T21:39:07.051566+0000","last_clean":"2024-04-04T21:39:07.051566+0000","last_became_active":"2024-04-04T21:03:49.734675+0000","last_became_peered":"2024-04-04T21:03:49.734675+0000","last_unstale":"2024-04-04T21:39:07.051566+0000","last_undegraded":"2024-04-04T21:39:07.051566+0000","last_fullsized":"2024-04-04T21:39:07.051566+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:02:53.025876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2333,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.063558+0000","last_change":"2024-04-04T21:03:46.411154+0000","last_active":"2024-04-04T21:39:07.063558+0000","last_peered":"2024-04-04T21:39:07.063558+0000","last_clean":"2024-04-04T21:39:07.063558+0000","last_became_active":"2024-04-04T21:03:46.400875+0000","last_became_peered":"2024-04-04T21:03:46.400875+0000","last_unstale":"2024-04-04T21:39:07.063558+0000","last_undegraded":"2024-04-04T21:39:07.063558+0000","last_fullsized":"2024-04-04T21:39:07.063558+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:55:55.301166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2287,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.458495+0000","last_change":"2024-04-04T21:03:47.549919+0000","last_active":"2024-04-04T21:39:08.458495+0000","last_peered":"2024-04-04T21:39:08.458495+0000","last_clean":"2024-04-04T21:39:08.458495+0000","last_became_active":"2024-04-04T21:03:47.514084+0000","last_became_peered":"2024-04-04T21:03:47.514084+0000","last_unstale":"2024-04-04T21:39:08.458495+0000","last_undegraded":"2024-04-04T21:39:08.458495+0000","last_fullsized":"2024-04-04T21:39:08.458495+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T01:39:02.880998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2332,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.048147+0000","last_change":"2024-04-04T21:03:51.865620+0000","last_active":"2024-04-04T21:39:07.048147+0000","last_peered":"2024-04-04T21:39:07.048147+0000","last_clean":"2024-04-04T21:39:07.048147+0000","last_became_active":"2024-04-04T21:03:51.847711+0000","last_became_peered":"2024-04-04T21:03:51.847711+0000","last_unstale":"2024-04-04T21:39:07.048147+0000","last_undegraded":"2024-04-04T21:39:07.048147+0000","last_fullsized":"2024-04-04T21:39:07.048147+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:06:45.768228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2295,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.226407+0000","last_change":"2024-04-04T21:03:40.329642+0000","last_active":"2024-04-04T21:39:08.226407+0000","last_peered":"2024-04-04T21:39:08.226407+0000","last_clean":"2024-04-04T21:39:08.226407+0000","last_became_active":"2024-04-04T21:03:40.322843+0000","last_became_peered":"2024-04-04T21:03:40.322843+0000","last_unstale":"2024-04-04T21:39:08.226407+0000","last_undegraded":"2024-04-04T21:39:08.226407+0000","last_fullsized":"2024-04-04T21:39:08.226407+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T05:04:33.425931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2306,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.379857+0000","last_change":"2024-04-04T21:03:39.843540+0000","last_active":"2024-04-04T21:39:08.379857+0000","last_peered":"2024-04-04T21:39:08.379857+0000","last_clean":"2024-04-04T21:39:08.379857+0000","last_became_active":"2024-04-04T21:03:39.803236+0000","last_became_peered":"2024-04-04T21:03:39.803236+0000","last_unstale":"2024-04-04T21:39:08.379857+0000","last_undegraded":"2024-04-04T21:39:08.379857+0000","last_fullsized":"2024-04-04T21:39:08.379857+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:06:04.364199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2295,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.166370+0000","last_change":"2024-04-04T21:03:40.013569+0000","last_active":"2024-04-04T21:39:08.166370+0000","last_peered":"2024-04-04T21:39:08.166370+0000","last_clean":"2024-04-04T21:39:08.166370+0000","last_became_active":"2024-04-04T21:03:40.004086+0000","last_became_peered":"2024-04-04T21:03:40.004086+0000","last_unstale":"2024-04-04T21:39:08.166370+0000","last_undegraded":"2024-04-04T21:39:08.166370+0000","last_fullsized":"2024-04-04T21:39:08.166370+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T23:37:38.542875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2310,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.229426+0000","last_change":"2024-04-04T21:03:43.781555+0000","last_active":"2024-04-04T21:39:09.229426+0000","last_peered":"2024-04-04T21:39:09.229426+0000","last_clean":"2024-04-04T21:39:09.229426+0000","last_became_active":"2024-04-04T21:03:43.766926+0000","last_became_peered":"2024-04-04T21:03:43.766926+0000","last_unstale":"2024-04-04T21:39:09.229426+0000","last_undegraded":"2024-04-04T21:39:09.229426+0000","last_fullsized":"2024-04-04T21:39:09.229426+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T06:42:24.367113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2283,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.123671+0000","last_change":"2024-04-04T21:03:43.156915+0000","last_active":"2024-04-04T21:39:07.123671+0000","last_peered":"2024-04-04T21:39:07.123671+0000","last_clean":"2024-04-04T21:39:07.123671+0000","last_became_active":"2024-04-04T21:03:43.143344+0000","last_became_peered":"2024-04-04T21:03:43.143344+0000","last_unstale":"2024-04-04T21:39:07.123671+0000","last_undegraded":"2024-04-04T21:39:07.123671+0000","last_fullsized":"2024-04-04T21:39:07.123671+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T22:48:08.738584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2304,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.368340+0000","last_change":"2024-04-04T21:03:42.394120+0000","last_active":"2024-04-04T21:39:08.368340+0000","last_peered":"2024-04-04T21:39:08.368340+0000","last_clean":"2024-04-04T21:39:08.368340+0000","last_became_active":"2024-04-04T21:03:42.379382+0000","last_became_peered":"2024-04-04T21:03:42.379382+0000","last_unstale":"2024-04-04T21:39:08.368340+0000","last_undegraded":"2024-04-04T21:39:08.368340+0000","last_fullsized":"2024-04-04T21:39:08.368340+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:09:54.791797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2288,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.424813+0000","last_change":"2024-04-04T21:03:47.554482+0000","last_active":"2024-04-04T21:39:08.424813+0000","last_peered":"2024-04-04T21:39:08.424813+0000","last_clean":"2024-04-04T21:39:08.424813+0000","last_became_active":"2024-04-04T21:03:47.524251+0000","last_became_peered":"2024-04-04T21:03:47.524251+0000","last_unstale":"2024-04-04T21:39:08.424813+0000","last_undegraded":"2024-04-04T21:39:08.424813+0000","last_fullsized":"2024-04-04T21:39:08.424813+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:11:26.408285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2308,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.252850+0000","last_change":"2024-04-04T21:03:45.814604+0000","last_active":"2024-04-04T21:39:09.252850+0000","last_peered":"2024-04-04T21:39:09.252850+0000","last_clean":"2024-04-04T21:39:09.252850+0000","last_became_active":"2024-04-04T21:03:45.808883+0000","last_became_peered":"2024-04-04T21:03:45.808883+0000","last_unstale":"2024-04-04T21:39:09.252850+0000","last_undegraded":"2024-04-04T21:39:09.252850+0000","last_fullsized":"2024-04-04T21:39:09.252850+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:56:32.831438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899346367,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68324,"kb_used_data":5204,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303516,"statfs":{"total":96636764160,"available":96566800384,"internally_reserved":0,"allocated":5328896,"data_stored":2606125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346367,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72484,"kb_used_data":5204,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299356,"statfs":{"total":96636764160,"available":96562540544,"internally_reserved":0,"allocated":5328896,"data_stored":2606125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346367,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72856,"kb_used_data":5580,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298984,"statfs":{"total":96636764160,"available":96562159616,"internally_reserved":0,"allocated":5713920,"data_stored":2996281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411777,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72484,"kb_used_data":5208,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299356,"statfs":{"total":96636764160,"available":96562540544,"internally_reserved":0,"allocated":5332992,"data_stored":2606130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509889,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72468,"kb_used_data":5188,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299372,"statfs":{"total":96636764160,"available":96562556928,"internally_reserved":0,"allocated":5312512,"data_stored":2598441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509884,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68300,"kb_used_data":5176,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303540,"statfs":{"total":96636764160,"available":96566824960,"internally_reserved":0,"allocated":5300224,"data_stored":2590993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542591,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72500,"kb_used_data":5220,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299340,"statfs":{"total":96636764160,"available":96562524160,"internally_reserved":0,"allocated":5345280,"data_stored":2612837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575293,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68712,"kb_used_data":5596,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303128,"statfs":{"total":96636764160,"available":96566403072,"internally_reserved":0,"allocated":5730304,"data_stored":3003965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-04T21:39:12.219 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-04T21:39:12.219 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-04T21:39:12.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.318+0000 7f70bd719640 10 monclient: get_monmap_and_config 2024-04-04T21:39:12.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.318+0000 7f70bd719640 10 monclient: build_initial_monmap 2024-04-04T21:39:12.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.318+0000 7f70bd719640 10 monclient: monmap: 2024-04-04T21:39:12.318 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:12.318 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:12.318 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:12.319277+0000 2024-04-04T21:39:12.318 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:12.319277+0000 2024-04-04T21:39:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:12.319 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:12.319 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:12.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.318+0000 7f70bd719640 1 Processor -- start 2024-04-04T21:39:12.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.318+0000 7f70bd719640 1 -- start start 2024-04-04T21:39:12.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.318+0000 7f70bd719640 10 monclient: init 2024-04-04T21:39:12.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:12.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-04-04T21:39:12.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f70b80a9a50 0x7f70b80a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 10 monclient(hunting): picked mon.noname-a con 0x7f70b80a9a50 addr v2:172.21.15.106:3300/0 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70b813f6a0 0x7f70b813fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 10 monclient(hunting): picked mon.noname-b con 0x7f70b813f6a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f70b8142880 0x7f70b8142d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 10 monclient(hunting): picked mon.noname-c con 0x7f70b8142880 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b80aa370 con 0x7f70b80a9a50 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b80ab750 con 0x7f70b8142880 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b8140120 con 0x7f70b813f6a0 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.319+0000 7f70bd719640 10 monclient(hunting): authenticate will time out at 3601.586623s 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b6ffd640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f70b80a9a50 0x7f70b80a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:12.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b67fc640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70b813f6a0 0x7f70b813fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b77fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f70b8142880 0x7f70b8142d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b6ffd640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f70b80a9a50 0x7f70b80a9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:60906/0 (socket says 172.21.15.106:60906) 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b6ffd640 1 -- 172.21.15.106:0/1442278699 learned_addr learned my addr 172.21.15.106:0/1442278699 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b6ffd640 10 monclient(hunting): get_auth_request con 0x7f70b80a9a50 auth_method 0 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b77fe640 10 monclient(hunting): get_auth_request con 0x7f70b8142880 auth_method 0 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b67fc640 10 monclient(hunting): get_auth_request con 0x7f70b813f6a0 auth_method 0 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b67fc640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.320+0000 7f70b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.321+0000 7f70b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:12.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.321+0000 7f70b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:12.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.321+0000 7f70b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:12.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.321+0000 7f70b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:12.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.321+0000 7f70b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:12.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.321+0000 7f70b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:12.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.321+0000 7f70b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:12.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.321+0000 7f70b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:12.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.321+0000 7f70b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:12.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.322+0000 7f70b77fe640 10 monclient(hunting): handle_auth_done global_id 22033 payload 274 2024-04-04T21:39:12.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.322+0000 7f70b77fe640 1 -- 172.21.15.106:0/1442278699 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70b813f6a0 msgr2=0x7f70b813fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:12.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.322+0000 7f70b77fe640 1 --2- 172.21.15.106:0/1442278699 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70b813f6a0 0x7f70b813fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.322+0000 7f70b77fe640 1 -- 172.21.15.106:0/1442278699 >> v2:172.21.15.106:3300/0 conn(0x7f70b80a9a50 msgr2=0x7f70b80a9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:12.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.322+0000 7f70b77fe640 1 --2- 172.21.15.106:0/1442278699 >> v2:172.21.15.106:3300/0 conn(0x7f70b80a9a50 0x7f70b80a9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.322+0000 7f70b77fe640 10 monclient: _finish_hunting 0 2024-04-04T21:39:12.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.322+0000 7f70b77fe640 1 monclient: found mon.noname-c 2024-04-04T21:39:12.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.322+0000 7f70b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:12.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.322+0000 7f70b77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:39:12.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.322+0000 7f70b77fe640 1 -- 172.21.15.106:0/1442278699 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b80aa660 con 0x7f70b8142880 2024-04-04T21:39:12.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.324+0000 7f70b77fe640 1 --2- 172.21.15.106:0/1442278699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b8142880 0x7f70b8142d10 secure :-1 s=READY pgs=5528 cs=0 l=1 rev1=1 crypto rx=0x7f70b80ab040 tx=0x7f70a802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc4978cc06101d4b server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:12.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70b5ffb640 1 -- 172.21.15.106:0/1442278699 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a8038070 con 0x7f70b8142880 2024-04-04T21:39:12.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:12.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70b5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:39:12.328 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70b5ffb640 10 monclient: dump: 2024-04-04T21:39:12.328 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:12.328 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:12.328 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.329 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.329 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:12.329 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:12.329 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:12.329 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:12.329 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:12.329 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:12.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70b5ffb640 10 monclient: _finish_auth 0 2024-04-04T21:39:12.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70b5ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:39:12.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:12.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70b5ffb640 1 -- 172.21.15.106:0/1442278699 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a8031700 con 0x7f70b8142880 2024-04-04T21:39:12.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70bd719640 5 monclient: authenticate success, global_id 22033 2024-04-04T21:39:12.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70bd719640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:39:12.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:12.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70bd719640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:12.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70b5ffb640 1 -- 172.21.15.106:0/1442278699 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a8031ad0 con 0x7f70b8142880 2024-04-04T21:39:12.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:12.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70bd719640 4 set_mon_vals no callback set 2024-04-04T21:39:12.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:12.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.328+0000 7f70b5ffb640 10 monclient: dump: 2024-04-04T21:39:12.330 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:12.330 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:12.330 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.330 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.330 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:12.330 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:12.330 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:12.330 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:12.331 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:12.331 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:12.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.329+0000 7f70bd719640 10 monclient: shutdown 2024-04-04T21:39:12.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.329+0000 7f70bd719640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:12.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.329+0000 7f70bd719640 1 -- 172.21.15.106:0/1442278699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b8142880 msgr2=0x7f70b8142d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:12.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.329+0000 7f70bd719640 1 --2- 172.21.15.106:0/1442278699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b8142880 0x7f70b8142d10 secure :-1 s=READY pgs=5528 cs=0 l=1 rev1=1 crypto rx=0x7f70b80ab040 tx=0x7f70a802b7a0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.329+0000 7f70bd719640 1 -- 172.21.15.106:0/1442278699 shutdown_connections 2024-04-04T21:39:12.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.329+0000 7f70bd719640 1 --2- 172.21.15.106:0/1442278699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b8142880 0x7f70b8142d10 unknown :-1 s=CLOSED pgs=5528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.329+0000 7f70bd719640 1 --2- 172.21.15.106:0/1442278699 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70b813f6a0 0x7f70b813fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.329+0000 7f70bd719640 1 --2- 172.21.15.106:0/1442278699 >> v2:172.21.15.106:3300/0 conn(0x7f70b80a9a50 0x7f70b80a9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.329+0000 7f70bd719640 1 -- 172.21.15.106:0/1442278699 >> 172.21.15.106:0/1442278699 conn(0x7f70b80b8030 msgr2=0x7f70b80b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:12.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.329+0000 7f70bd719640 1 -- 172.21.15.106:0/1442278699 shutdown_connections 2024-04-04T21:39:12.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.329+0000 7f70bd719640 1 -- 172.21.15.106:0/1442278699 wait complete. 2024-04-04T21:39:12.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.329+0000 7f70bd719640 10 monclient: build_initial_monmap 2024-04-04T21:39:12.335 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.329+0000 7f70bd719640 10 monclient: monmap: 2024-04-04T21:39:12.335 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:12.335 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:12.335 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:12.335 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:12.335 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:12.335 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:12.335 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:12.336 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:12.336 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:12.336 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:12.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 1 librados: starting msgr at 2024-04-04T21:39:12.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 1 librados: starting objecter 2024-04-04T21:39:12.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 1 Processor -- start 2024-04-04T21:39:12.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 1 -- start start 2024-04-04T21:39:12.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 1 librados: setting wanted keys 2024-04-04T21:39:12.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 1 librados: calling monclient init 2024-04-04T21:39:12.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 10 monclient: init 2024-04-04T21:39:12.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:12.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-04T21:39:12.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80a9a50 0x7f70b80ac510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:12.336 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 10 monclient(hunting): picked mon.noname-b con 0x7f70b80a9a50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:12.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70b813f6a0 0x7f70b813a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:12.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 10 monclient(hunting): picked mon.noname-c con 0x7f70b813f6a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:12.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70b8142880 0x7f70b813a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:12.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 10 monclient(hunting): picked mon.noname-a con 0x7f70b8142880 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:12.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:12.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b8145b60 con 0x7f70b8142880 2024-04-04T21:39:12.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:12.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b8143460 con 0x7f70b80a9a50 2024-04-04T21:39:12.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:12.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b8145ce0 con 0x7f70b813f6a0 2024-04-04T21:39:12.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:12.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.330+0000 7f70bd719640 10 monclient(hunting): authenticate will time out at 3601.598172s 2024-04-04T21:39:12.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.331+0000 7f70b77fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70b8142880 0x7f70b813a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:12.337 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.331+0000 7f70b67fc640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70b813f6a0 0x7f70b813a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:12.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.331+0000 7f70b6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80a9a50 0x7f70b80ac510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:12.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.331+0000 7f70b77fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70b8142880 0x7f70b813a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:60916/0 (socket says 172.21.15.106:60916) 2024-04-04T21:39:12.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.331+0000 7f70b77fe640 1 -- 172.21.15.106:0/494048290 learned_addr learned my addr 172.21.15.106:0/494048290 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:12.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.331+0000 7f70b77fe640 10 monclient(hunting): get_auth_request con 0x7f70b8142880 auth_method 0 2024-04-04T21:39:12.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.331+0000 7f70b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:12.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.331+0000 7f70b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:12.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.331+0000 7f70b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:12.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.331+0000 7f70b67fc640 10 monclient(hunting): get_auth_request con 0x7f70b813f6a0 auth_method 0 2024-04-04T21:39:12.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.331+0000 7f70b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:12.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.331+0000 7f70b67fc640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:12.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.331+0000 7f70b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:12.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.332+0000 7f70b6ffd640 10 monclient(hunting): get_auth_request con 0x7f70b80a9a50 auth_method 0 2024-04-04T21:39:12.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.332+0000 7f70b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:12.338 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.332+0000 7f70b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:12.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.332+0000 7f70b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:12.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.332+0000 7f70b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:12.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.332+0000 7f70b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:12.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.332+0000 7f70b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:12.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.332+0000 7f70b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:12.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.332+0000 7f70b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:12.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.332+0000 7f70b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:12.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.332+0000 7f70b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:12.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.332+0000 7f70b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:12.339 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.332+0000 7f70b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:12.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.335+0000 7f70b6ffd640 10 monclient(hunting): handle_auth_done global_id 22036 payload 824 2024-04-04T21:39:12.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.335+0000 7f70b6ffd640 1 -- 172.21.15.106:0/494048290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70b813f6a0 msgr2=0x7f70b813a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:12.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.335+0000 7f70b6ffd640 1 --2- 172.21.15.106:0/494048290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70b813f6a0 0x7f70b813a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.335+0000 7f70b6ffd640 1 -- 172.21.15.106:0/494048290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70b8142880 msgr2=0x7f70b813a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:12.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.335+0000 7f70b6ffd640 1 --2- 172.21.15.106:0/494048290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70b8142880 0x7f70b813a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.335+0000 7f70b6ffd640 10 monclient: _finish_hunting 0 2024-04-04T21:39:12.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.335+0000 7f70b6ffd640 1 monclient: found mon.noname-b 2024-04-04T21:39:12.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.335+0000 7f70b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:12.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.335+0000 7f70b6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:12.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.335+0000 7f70b6ffd640 1 -- 172.21.15.106:0/494048290 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b813ae80 con 0x7f70b80a9a50 2024-04-04T21:39:12.342 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.337+0000 7f70b6ffd640 1 --2- 172.21.15.106:0/494048290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80a9a50 0x7f70b80ac510 secure :-1 s=READY pgs=5529 cs=0 l=1 rev1=1 crypto rx=0x7f70b0009db0 tx=0x7f70b000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.337+0000 7f709f7fe640 10 client.?.objecter ms_handle_connect 0x7f70b80a9a50 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.337+0000 7f709f7fe640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f709f7fe640 1 -- 172.21.15.106:0/494048290 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b000cda0 con 0x7f70b80a9a50 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f709f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f709f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f709f7fe640 10 monclient: dump: 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:12.343 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:12.344 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:12.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f709f7fe640 10 monclient: _finish_auth 0 2024-04-04T21:39:12.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f709f7fe640 10 monclient: _check_auth_tickets 2024-04-04T21:39:12.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f709f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:12.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f709f7fe640 1 -- 172.21.15.106:0/494048290 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b0019070 con 0x7f70b80a9a50 2024-04-04T21:39:12.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f70bd719640 5 monclient: authenticate success, global_id 22036 2024-04-04T21:39:12.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f70bd719640 10 monclient: _renew_subs 2024-04-04T21:39:12.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f70bd719640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:12.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f70bd719640 1 -- 172.21.15.106:0/494048290 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b80ad770 con 0x7f70b80a9a50 2024-04-04T21:39:12.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f709f7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:12.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f709f7fe640 1 -- 172.21.15.106:0/494048290 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b0014680 con 0x7f70b80a9a50 2024-04-04T21:39:12.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f709f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:12.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f709f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:12.344 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f709f7fe640 10 monclient: dump: 2024-04-04T21:39:12.345 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:12.345 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:12.345 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.345 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.345 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:12.345 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:12.345 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:12.345 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:12.345 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:12.345 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:12.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f70b57fa640 4 set_mon_vals no callback set 2024-04-04T21:39:12.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f70bd719640 10 client.22036.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:12.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f70bd719640 10 monclient: _renew_subs 2024-04-04T21:39:12.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f70bd719640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:12.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f70bd719640 1 -- 172.21.15.106:0/494048290 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b80ad250 con 0x7f70b80a9a50 2024-04-04T21:39:12.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.340+0000 7f70bd719640 1 librados: init done 2024-04-04T21:39:12.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.341+0000 7f70bd719640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:12.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.341+0000 7f70bd719640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:12.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.341+0000 7f70bd719640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:12.346 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.341+0000 7f70bd719640 1 -- 172.21.15.106:0/494048290 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b804f050 con 0x7f70b80a9a50 2024-04-04T21:39:12.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.360+0000 7f709f7fe640 1 -- 172.21.15.106:0/494048290 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f70b0005d10 con 0x7f70b80a9a50 2024-04-04T21:39:12.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.360+0000 7f709f7fe640 1 --2- 172.21.15.106:0/494048290 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f70a406ce60 0x7f70a406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:12.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.360+0000 7f70b67fc640 1 --2- 172.21.15.106:0/494048290 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f70a406ce60 0x7f70a406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:12.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.360+0000 7f70b67fc640 10 monclient: get_auth_request con 0x7f70a406ce60 auth_method 0 2024-04-04T21:39:12.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.361+0000 7f709f7fe640 10 client.22036.objecter ms_handle_connect 0x7f70a406ce60 2024-04-04T21:39:12.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.361+0000 7f70b67fc640 1 --2- 172.21.15.106:0/494048290 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f70a406ce60 0x7f70a406f320 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f70b813f0a0 tx=0x7f70ac009190 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:12.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.362+0000 7f709f7fe640 1 -- 172.21.15.106:0/494048290 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f70b0057a10 con 0x7f70b80a9a50 2024-04-04T21:39:12.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.362+0000 7f709f7fe640 10 client.22036.objecter ms_dispatch 0x7f70b8000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:12.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.362+0000 7f709f7fe640 3 client.22036.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:12.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.362+0000 7f709f7fe640 3 client.22036.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:12.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.362+0000 7f709f7fe640 20 client.22036.objecter dump_active .. 0 homeless 2024-04-04T21:39:12.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.603+0000 7f709f7fe640 1 -- 172.21.15.106:0/494048290 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f70b0010040 con 0x7f70b80a9a50 2024-04-04T21:39:12.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.603+0000 7f709f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:12.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.603+0000 7f709f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:12.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.779+0000 7f70bd719640 1 -- 172.21.15.106:0/494048290 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f70b813ec60 con 0x7f70a406ce60 2024-04-04T21:39:12.782 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.782+0000 7f709f7fe640 1 -- 172.21.15.106:0/494048290 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+116264 (secure 0 0 0) 0x7f70b813ec60 con 0x7f70a406ce60 2024-04-04T21:39:12.783 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:39:12.784 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-04-04T21:39:12.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.784+0000 7f70bd719640 10 librados: watch_flush enter 2024-04-04T21:39:12.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.784+0000 7f70bd719640 10 librados: watch_flush exit 2024-04-04T21:39:12.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.784+0000 7f70bd719640 20 client.22036.objecter shutdown clearing up homeless session... 2024-04-04T21:39:12.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.784+0000 7f70bd719640 10 client.22036.objecter successfully canceled tick 2024-04-04T21:39:12.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 1 -- 172.21.15.106:0/494048290 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f70a406ce60 msgr2=0x7f70a406f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:12.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 1 --2- 172.21.15.106:0/494048290 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f70a406ce60 0x7f70a406f320 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f70b813f0a0 tx=0x7f70ac009190 comp rx=0 tx=0).stop 2024-04-04T21:39:12.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 10 monclient: shutdown 2024-04-04T21:39:12.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:12.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 1 -- 172.21.15.106:0/494048290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80a9a50 msgr2=0x7f70b80ac510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:12.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 1 --2- 172.21.15.106:0/494048290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80a9a50 0x7f70b80ac510 secure :-1 s=READY pgs=5529 cs=0 l=1 rev1=1 crypto rx=0x7f70b0009db0 tx=0x7f70b000c490 comp rx=0 tx=0).stop 2024-04-04T21:39:12.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 1 -- 172.21.15.106:0/494048290 shutdown_connections 2024-04-04T21:39:12.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 1 --2- 172.21.15.106:0/494048290 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f70a406ce60 0x7f70a406f320 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 1 --2- 172.21.15.106:0/494048290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70b8142880 0x7f70b813a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 1 --2- 172.21.15.106:0/494048290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70b813f6a0 0x7f70b813a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 1 --2- 172.21.15.106:0/494048290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80a9a50 0x7f70b80ac510 unknown :-1 s=CLOSED pgs=5529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 1 -- 172.21.15.106:0/494048290 >> 172.21.15.106:0/494048290 conn(0x7f70b80b8030 msgr2=0x7f70b80b77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:12.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 1 -- 172.21.15.106:0/494048290 shutdown_connections 2024-04-04T21:39:12.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 1 -- 172.21.15.106:0/494048290 wait complete. 2024-04-04T21:39:12.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.785+0000 7f70bd719640 1 librados: shutdown 2024-04-04T21:39:12.796 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":2312,"stamp":"2024-04-04T21:39:11.059877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":87,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":568128,"kb_used_data":42376,"kb_used_omap":12,"kb_used_meta":525619,"kb_avail":754406592,"statfs":{"total":773094113280,"available":772512350208,"internally_reserved":0,"allocated":43393024,"data_stored":21620897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":538234417},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":158,"apply_latency_ms":158,"commit_latency_ns":158000000,"apply_latency_ns":158000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.996579"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2332,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.039530+0000","last_change":"2024-04-04T21:03:54.942652+0000","last_active":"2024-04-04T21:39:07.039530+0000","last_peered":"2024-04-04T21:39:07.039530+0000","last_clean":"2024-04-04T21:39:07.039530+0000","last_became_active":"2024-04-04T21:03:54.933965+0000","last_became_peered":"2024-04-04T21:03:54.933965+0000","last_unstale":"2024-04-04T21:39:07.039530+0000","last_undegraded":"2024-04-04T21:39:07.039530+0000","last_fullsized":"2024-04-04T21:39:07.039530+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:47:55.727002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2305,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.360894+0000","last_change":"2024-04-04T21:03:44.565986+0000","last_active":"2024-04-04T21:39:08.360894+0000","last_peered":"2024-04-04T21:39:08.360894+0000","last_clean":"2024-04-04T21:39:08.360894+0000","last_became_active":"2024-04-04T21:03:44.551107+0000","last_became_peered":"2024-04-04T21:03:44.551107+0000","last_unstale":"2024-04-04T21:39:08.360894+0000","last_undegraded":"2024-04-04T21:39:08.360894+0000","last_fullsized":"2024-04-04T21:39:08.360894+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:51:23.943936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2301,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.405327+0000","last_change":"2024-04-04T21:03:52.793808+0000","last_active":"2024-04-04T21:39:08.405327+0000","last_peered":"2024-04-04T21:39:08.405327+0000","last_clean":"2024-04-04T21:39:08.405327+0000","last_became_active":"2024-04-04T21:03:52.772128+0000","last_became_peered":"2024-04-04T21:03:52.772128+0000","last_unstale":"2024-04-04T21:39:08.405327+0000","last_undegraded":"2024-04-04T21:39:08.405327+0000","last_fullsized":"2024-04-04T21:39:08.405327+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:44:28.885034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2287,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.407788+0000","last_change":"2024-04-04T21:03:41.361551+0000","last_active":"2024-04-04T21:39:08.407788+0000","last_peered":"2024-04-04T21:39:08.407788+0000","last_clean":"2024-04-04T21:39:08.407788+0000","last_became_active":"2024-04-04T21:03:41.346161+0000","last_became_peered":"2024-04-04T21:03:41.346161+0000","last_unstale":"2024-04-04T21:39:08.407788+0000","last_undegraded":"2024-04-04T21:39:08.407788+0000","last_fullsized":"2024-04-04T21:39:08.407788+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:14:45.723197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2289,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.440013+0000","last_change":"2024-04-04T21:03:42.130175+0000","last_active":"2024-04-04T21:39:08.440013+0000","last_peered":"2024-04-04T21:39:08.440013+0000","last_clean":"2024-04-04T21:39:08.440013+0000","last_became_active":"2024-04-04T21:03:42.122180+0000","last_became_peered":"2024-04-04T21:03:42.122180+0000","last_unstale":"2024-04-04T21:39:08.440013+0000","last_undegraded":"2024-04-04T21:39:08.440013+0000","last_fullsized":"2024-04-04T21:39:08.440013+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:01:39.698809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2318,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.250973+0000","last_change":"2024-04-04T21:03:42.949631+0000","last_active":"2024-04-04T21:39:09.250973+0000","last_peered":"2024-04-04T21:39:09.250973+0000","last_clean":"2024-04-04T21:39:09.250973+0000","last_became_active":"2024-04-04T21:03:42.940728+0000","last_became_peered":"2024-04-04T21:03:42.940728+0000","last_unstale":"2024-04-04T21:39:09.250973+0000","last_undegraded":"2024-04-04T21:39:09.250973+0000","last_fullsized":"2024-04-04T21:39:09.250973+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:24:58.260325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2306,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.372867+0000","last_change":"2024-04-04T21:03:43.791130+0000","last_active":"2024-04-04T21:39:08.372867+0000","last_peered":"2024-04-04T21:39:08.372867+0000","last_clean":"2024-04-04T21:39:08.372867+0000","last_became_active":"2024-04-04T21:03:43.596620+0000","last_became_peered":"2024-04-04T21:03:43.596620+0000","last_unstale":"2024-04-04T21:39:08.372867+0000","last_undegraded":"2024-04-04T21:39:08.372867+0000","last_fullsized":"2024-04-04T21:39:08.372867+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T21:23:20.072459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2307,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.377536+0000","last_change":"2024-04-04T21:03:52.362151+0000","last_active":"2024-04-04T21:39:08.377536+0000","last_peered":"2024-04-04T21:39:08.377536+0000","last_clean":"2024-04-04T21:39:08.377536+0000","last_became_active":"2024-04-04T21:03:52.351392+0000","last_became_peered":"2024-04-04T21:03:52.351392+0000","last_unstale":"2024-04-04T21:39:08.377536+0000","last_undegraded":"2024-04-04T21:39:08.377536+0000","last_fullsized":"2024-04-04T21:39:08.377536+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T01:47:25.888369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"39'7","reported_seq":2332,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.239999+0000","last_change":"2024-04-04T21:03:50.565469+0000","last_active":"2024-04-04T21:39:09.239999+0000","last_peered":"2024-04-04T21:39:09.239999+0000","last_clean":"2024-04-04T21:39:09.239999+0000","last_became_active":"2024-04-04T21:03:50.287197+0000","last_became_peered":"2024-04-04T21:03:50.287197+0000","last_unstale":"2024-04-04T21:39:09.239999+0000","last_undegraded":"2024-04-04T21:39:09.239999+0000","last_fullsized":"2024-04-04T21:39:09.239999+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T04:15:05.649671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2290,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.366167+0000","last_change":"2024-04-04T21:03:44.924431+0000","last_active":"2024-04-04T21:39:08.366167+0000","last_peered":"2024-04-04T21:39:08.366167+0000","last_clean":"2024-04-04T21:39:08.366167+0000","last_became_active":"2024-04-04T21:03:44.913048+0000","last_became_peered":"2024-04-04T21:03:44.913048+0000","last_unstale":"2024-04-04T21:39:08.366167+0000","last_undegraded":"2024-04-04T21:39:08.366167+0000","last_fullsized":"2024-04-04T21:39:08.366167+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T22:52:04.454928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2286,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.112318+0000","last_change":"2024-04-04T21:03:44.496009+0000","last_active":"2024-04-04T21:39:07.112318+0000","last_peered":"2024-04-04T21:39:07.112318+0000","last_clean":"2024-04-04T21:39:07.112318+0000","last_became_active":"2024-04-04T21:03:44.451112+0000","last_became_peered":"2024-04-04T21:03:44.451112+0000","last_unstale":"2024-04-04T21:39:07.112318+0000","last_undegraded":"2024-04-04T21:39:07.112318+0000","last_fullsized":"2024-04-04T21:39:07.112318+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:12:30.730952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2334,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.024282+0000","last_change":"2024-04-04T21:03:44.379999+0000","last_active":"2024-04-04T21:39:07.024282+0000","last_peered":"2024-04-04T21:39:07.024282+0000","last_clean":"2024-04-04T21:39:07.024282+0000","last_became_active":"2024-04-04T21:03:44.366452+0000","last_became_peered":"2024-04-04T21:03:44.366452+0000","last_unstale":"2024-04-04T21:39:07.024282+0000","last_undegraded":"2024-04-04T21:39:07.024282+0000","last_fullsized":"2024-04-04T21:39:07.024282+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:21:39.463272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":2318,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.168312+0000","last_change":"2024-04-04T21:02:20.509950+0000","last_active":"2024-04-04T21:39:08.168312+0000","last_peered":"2024-04-04T21:39:08.168312+0000","last_clean":"2024-04-04T21:39:08.168312+0000","last_became_active":"2024-04-04T21:02:19.876220+0000","last_became_peered":"2024-04-04T21:02:19.876220+0000","last_unstale":"2024-04-04T21:39:08.168312+0000","last_undegraded":"2024-04-04T21:39:08.168312+0000","last_fullsized":"2024-04-04T21:39:08.168312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:05.463818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:05.463818+0000","last_clean_scrub_stamp":"2024-04-04T21:02:05.463818+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:18:20.734360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2323,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.257109+0000","last_change":"2024-04-04T21:02:24.476744+0000","last_active":"2024-04-04T21:39:09.257109+0000","last_peered":"2024-04-04T21:39:09.257109+0000","last_clean":"2024-04-04T21:39:09.257109+0000","last_became_active":"2024-04-04T21:02:24.466203+0000","last_became_peered":"2024-04-04T21:02:24.466203+0000","last_unstale":"2024-04-04T21:39:09.257109+0000","last_undegraded":"2024-04-04T21:39:09.257109+0000","last_fullsized":"2024-04-04T21:39:09.257109+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T21:40:50.021063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2289,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.369121+0000","last_change":"2024-04-04T21:03:40.772628+0000","last_active":"2024-04-04T21:39:08.369121+0000","last_peered":"2024-04-04T21:39:08.369121+0000","last_clean":"2024-04-04T21:39:08.369121+0000","last_became_active":"2024-04-04T21:03:40.758544+0000","last_became_peered":"2024-04-04T21:03:40.758544+0000","last_unstale":"2024-04-04T21:39:08.369121+0000","last_undegraded":"2024-04-04T21:39:08.369121+0000","last_fullsized":"2024-04-04T21:39:08.369121+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:22:11.238559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2324,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.235421+0000","last_change":"2024-04-04T21:02:24.174900+0000","last_active":"2024-04-04T21:39:09.235421+0000","last_peered":"2024-04-04T21:39:09.235421+0000","last_clean":"2024-04-04T21:39:09.235421+0000","last_became_active":"2024-04-04T21:02:24.146676+0000","last_became_peered":"2024-04-04T21:02:24.146676+0000","last_unstale":"2024-04-04T21:39:09.235421+0000","last_undegraded":"2024-04-04T21:39:09.235421+0000","last_fullsized":"2024-04-04T21:39:09.235421+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:47:28.827956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2287,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.411575+0000","last_change":"2024-04-04T21:03:39.388042+0000","last_active":"2024-04-04T21:39:08.411575+0000","last_peered":"2024-04-04T21:39:08.411575+0000","last_clean":"2024-04-04T21:39:08.411575+0000","last_became_active":"2024-04-04T21:03:39.382089+0000","last_became_peered":"2024-04-04T21:03:39.382089+0000","last_unstale":"2024-04-04T21:39:08.411575+0000","last_undegraded":"2024-04-04T21:39:08.411575+0000","last_fullsized":"2024-04-04T21:39:08.411575+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T01:47:50.490851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2330,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.242110+0000","last_change":"2024-04-04T21:02:24.189161+0000","last_active":"2024-04-04T21:39:09.242110+0000","last_peered":"2024-04-04T21:39:09.242110+0000","last_clean":"2024-04-04T21:39:09.242110+0000","last_became_active":"2024-04-04T21:02:24.159102+0000","last_became_peered":"2024-04-04T21:02:24.159102+0000","last_unstale":"2024-04-04T21:39:09.242110+0000","last_undegraded":"2024-04-04T21:39:09.242110+0000","last_fullsized":"2024-04-04T21:39:09.242110+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:02:24.748876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2296,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.220163+0000","last_change":"2024-04-04T21:03:45.245420+0000","last_active":"2024-04-04T21:39:08.220163+0000","last_peered":"2024-04-04T21:39:08.220163+0000","last_clean":"2024-04-04T21:39:08.220163+0000","last_became_active":"2024-04-04T21:03:45.238093+0000","last_became_peered":"2024-04-04T21:03:45.238093+0000","last_unstale":"2024-04-04T21:39:08.220163+0000","last_undegraded":"2024-04-04T21:39:08.220163+0000","last_fullsized":"2024-04-04T21:39:08.220163+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T06:46:05.953419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2323,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.232461+0000","last_change":"2024-04-04T21:02:24.359272+0000","last_active":"2024-04-04T21:39:09.232461+0000","last_peered":"2024-04-04T21:39:09.232461+0000","last_clean":"2024-04-04T21:39:09.232461+0000","last_became_active":"2024-04-04T21:02:24.352103+0000","last_became_peered":"2024-04-04T21:02:24.352103+0000","last_unstale":"2024-04-04T21:39:09.232461+0000","last_undegraded":"2024-04-04T21:39:09.232461+0000","last_fullsized":"2024-04-04T21:39:09.232461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:36:54.774282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2289,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.382830+0000","last_change":"2024-04-04T21:03:42.104012+0000","last_active":"2024-04-04T21:39:08.382830+0000","last_peered":"2024-04-04T21:39:08.382830+0000","last_clean":"2024-04-04T21:39:08.382830+0000","last_became_active":"2024-04-04T21:03:42.098166+0000","last_became_peered":"2024-04-04T21:03:42.098166+0000","last_unstale":"2024-04-04T21:39:08.382830+0000","last_undegraded":"2024-04-04T21:39:08.382830+0000","last_fullsized":"2024-04-04T21:39:08.382830+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:37:16.291707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2300,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.460408+0000","last_change":"2024-04-04T21:02:24.374143+0000","last_active":"2024-04-04T21:39:08.460408+0000","last_peered":"2024-04-04T21:39:08.460408+0000","last_clean":"2024-04-04T21:39:08.460408+0000","last_became_active":"2024-04-04T21:02:24.359409+0000","last_became_peered":"2024-04-04T21:02:24.359409+0000","last_unstale":"2024-04-04T21:39:08.460408+0000","last_undegraded":"2024-04-04T21:39:08.460408+0000","last_fullsized":"2024-04-04T21:39:08.460408+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:43:28.396677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2305,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.370162+0000","last_change":"2024-04-04T21:03:42.389368+0000","last_active":"2024-04-04T21:39:08.370162+0000","last_peered":"2024-04-04T21:39:08.370162+0000","last_clean":"2024-04-04T21:39:08.370162+0000","last_became_active":"2024-04-04T21:03:42.371847+0000","last_became_peered":"2024-04-04T21:03:42.371847+0000","last_unstale":"2024-04-04T21:39:08.370162+0000","last_undegraded":"2024-04-04T21:39:08.370162+0000","last_fullsized":"2024-04-04T21:39:08.370162+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T06:52:54.899883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2322,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.262941+0000","last_change":"2024-04-04T21:02:24.241805+0000","last_active":"2024-04-04T21:39:09.262941+0000","last_peered":"2024-04-04T21:39:09.262941+0000","last_clean":"2024-04-04T21:39:09.262941+0000","last_became_active":"2024-04-04T21:02:24.236662+0000","last_became_peered":"2024-04-04T21:02:24.236662+0000","last_unstale":"2024-04-04T21:39:09.262941+0000","last_undegraded":"2024-04-04T21:39:09.262941+0000","last_fullsized":"2024-04-04T21:39:09.262941+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:47:33.277018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2306,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.375694+0000","last_change":"2024-04-04T21:03:48.545083+0000","last_active":"2024-04-04T21:39:08.375694+0000","last_peered":"2024-04-04T21:39:08.375694+0000","last_clean":"2024-04-04T21:39:08.375694+0000","last_became_active":"2024-04-04T21:03:48.539598+0000","last_became_peered":"2024-04-04T21:03:48.539598+0000","last_unstale":"2024-04-04T21:39:08.375694+0000","last_undegraded":"2024-04-04T21:39:08.375694+0000","last_fullsized":"2024-04-04T21:39:08.375694+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:16:56.765838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2313,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.389560+0000","last_change":"2024-04-04T21:02:25.438251+0000","last_active":"2024-04-04T21:39:08.389560+0000","last_peered":"2024-04-04T21:39:08.389560+0000","last_clean":"2024-04-04T21:39:08.389560+0000","last_became_active":"2024-04-04T21:02:25.412507+0000","last_became_peered":"2024-04-04T21:02:25.412507+0000","last_unstale":"2024-04-04T21:39:08.389560+0000","last_undegraded":"2024-04-04T21:39:08.389560+0000","last_fullsized":"2024-04-04T21:39:08.389560+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:17:04.580586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2309,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.261110+0000","last_change":"2024-04-04T21:03:56.049841+0000","last_active":"2024-04-04T21:39:09.261110+0000","last_peered":"2024-04-04T21:39:09.261110+0000","last_clean":"2024-04-04T21:39:09.261110+0000","last_became_active":"2024-04-04T21:03:56.035576+0000","last_became_peered":"2024-04-04T21:03:56.035576+0000","last_unstale":"2024-04-04T21:39:09.261110+0000","last_undegraded":"2024-04-04T21:39:09.261110+0000","last_fullsized":"2024-04-04T21:39:09.261110+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:29:26.354814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2321,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.264847+0000","last_change":"2024-04-04T21:02:25.403337+0000","last_active":"2024-04-04T21:39:09.264847+0000","last_peered":"2024-04-04T21:39:09.264847+0000","last_clean":"2024-04-04T21:39:09.264847+0000","last_became_active":"2024-04-04T21:02:25.394276+0000","last_became_peered":"2024-04-04T21:02:25.394276+0000","last_unstale":"2024-04-04T21:39:09.264847+0000","last_undegraded":"2024-04-04T21:39:09.264847+0000","last_fullsized":"2024-04-04T21:39:09.264847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:42:02.653040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2316,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.248209+0000","last_change":"2024-04-04T21:03:55.487132+0000","last_active":"2024-04-04T21:39:09.248209+0000","last_peered":"2024-04-04T21:39:09.248209+0000","last_clean":"2024-04-04T21:39:09.248209+0000","last_became_active":"2024-04-04T21:03:55.479649+0000","last_became_peered":"2024-04-04T21:03:55.479649+0000","last_unstale":"2024-04-04T21:39:09.248209+0000","last_undegraded":"2024-04-04T21:39:09.248209+0000","last_fullsized":"2024-04-04T21:39:09.248209+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:54:34.290339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2333,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.051566+0000","last_change":"2024-04-04T21:03:49.740972+0000","last_active":"2024-04-04T21:39:07.051566+0000","last_peered":"2024-04-04T21:39:07.051566+0000","last_clean":"2024-04-04T21:39:07.051566+0000","last_became_active":"2024-04-04T21:03:49.734675+0000","last_became_peered":"2024-04-04T21:03:49.734675+0000","last_unstale":"2024-04-04T21:39:07.051566+0000","last_undegraded":"2024-04-04T21:39:07.051566+0000","last_fullsized":"2024-04-04T21:39:07.051566+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:02:53.025876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2333,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.063558+0000","last_change":"2024-04-04T21:03:46.411154+0000","last_active":"2024-04-04T21:39:07.063558+0000","last_peered":"2024-04-04T21:39:07.063558+0000","last_clean":"2024-04-04T21:39:07.063558+0000","last_became_active":"2024-04-04T21:03:46.400875+0000","last_became_peered":"2024-04-04T21:03:46.400875+0000","last_unstale":"2024-04-04T21:39:07.063558+0000","last_undegraded":"2024-04-04T21:39:07.063558+0000","last_fullsized":"2024-04-04T21:39:07.063558+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:55:55.301166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2287,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.458495+0000","last_change":"2024-04-04T21:03:47.549919+0000","last_active":"2024-04-04T21:39:08.458495+0000","last_peered":"2024-04-04T21:39:08.458495+0000","last_clean":"2024-04-04T21:39:08.458495+0000","last_became_active":"2024-04-04T21:03:47.514084+0000","last_became_peered":"2024-04-04T21:03:47.514084+0000","last_unstale":"2024-04-04T21:39:08.458495+0000","last_undegraded":"2024-04-04T21:39:08.458495+0000","last_fullsized":"2024-04-04T21:39:08.458495+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T01:39:02.880998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2332,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.048147+0000","last_change":"2024-04-04T21:03:51.865620+0000","last_active":"2024-04-04T21:39:07.048147+0000","last_peered":"2024-04-04T21:39:07.048147+0000","last_clean":"2024-04-04T21:39:07.048147+0000","last_became_active":"2024-04-04T21:03:51.847711+0000","last_became_peered":"2024-04-04T21:03:51.847711+0000","last_unstale":"2024-04-04T21:39:07.048147+0000","last_undegraded":"2024-04-04T21:39:07.048147+0000","last_fullsized":"2024-04-04T21:39:07.048147+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:06:45.768228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2295,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.226407+0000","last_change":"2024-04-04T21:03:40.329642+0000","last_active":"2024-04-04T21:39:08.226407+0000","last_peered":"2024-04-04T21:39:08.226407+0000","last_clean":"2024-04-04T21:39:08.226407+0000","last_became_active":"2024-04-04T21:03:40.322843+0000","last_became_peered":"2024-04-04T21:03:40.322843+0000","last_unstale":"2024-04-04T21:39:08.226407+0000","last_undegraded":"2024-04-04T21:39:08.226407+0000","last_fullsized":"2024-04-04T21:39:08.226407+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T05:04:33.425931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2306,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.379857+0000","last_change":"2024-04-04T21:03:39.843540+0000","last_active":"2024-04-04T21:39:08.379857+0000","last_peered":"2024-04-04T21:39:08.379857+0000","last_clean":"2024-04-04T21:39:08.379857+0000","last_became_active":"2024-04-04T21:03:39.803236+0000","last_became_peered":"2024-04-04T21:03:39.803236+0000","last_unstale":"2024-04-04T21:39:08.379857+0000","last_undegraded":"2024-04-04T21:39:08.379857+0000","last_fullsized":"2024-04-04T21:39:08.379857+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:06:04.364199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2295,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.166370+0000","last_change":"2024-04-04T21:03:40.013569+0000","last_active":"2024-04-04T21:39:08.166370+0000","last_peered":"2024-04-04T21:39:08.166370+0000","last_clean":"2024-04-04T21:39:08.166370+0000","last_became_active":"2024-04-04T21:03:40.004086+0000","last_became_peered":"2024-04-04T21:03:40.004086+0000","last_unstale":"2024-04-04T21:39:08.166370+0000","last_undegraded":"2024-04-04T21:39:08.166370+0000","last_fullsized":"2024-04-04T21:39:08.166370+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T23:37:38.542875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2310,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.229426+0000","last_change":"2024-04-04T21:03:43.781555+0000","last_active":"2024-04-04T21:39:09.229426+0000","last_peered":"2024-04-04T21:39:09.229426+0000","last_clean":"2024-04-04T21:39:09.229426+0000","last_became_active":"2024-04-04T21:03:43.766926+0000","last_became_peered":"2024-04-04T21:03:43.766926+0000","last_unstale":"2024-04-04T21:39:09.229426+0000","last_undegraded":"2024-04-04T21:39:09.229426+0000","last_fullsized":"2024-04-04T21:39:09.229426+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T06:42:24.367113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2283,"reported_epoch":1193,"state":"active+clean","last_fresh":"2024-04-04T21:39:07.123671+0000","last_change":"2024-04-04T21:03:43.156915+0000","last_active":"2024-04-04T21:39:07.123671+0000","last_peered":"2024-04-04T21:39:07.123671+0000","last_clean":"2024-04-04T21:39:07.123671+0000","last_became_active":"2024-04-04T21:03:43.143344+0000","last_became_peered":"2024-04-04T21:03:43.143344+0000","last_unstale":"2024-04-04T21:39:07.123671+0000","last_undegraded":"2024-04-04T21:39:07.123671+0000","last_fullsized":"2024-04-04T21:39:07.123671+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T22:48:08.738584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2304,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.368340+0000","last_change":"2024-04-04T21:03:42.394120+0000","last_active":"2024-04-04T21:39:08.368340+0000","last_peered":"2024-04-04T21:39:08.368340+0000","last_clean":"2024-04-04T21:39:08.368340+0000","last_became_active":"2024-04-04T21:03:42.379382+0000","last_became_peered":"2024-04-04T21:03:42.379382+0000","last_unstale":"2024-04-04T21:39:08.368340+0000","last_undegraded":"2024-04-04T21:39:08.368340+0000","last_fullsized":"2024-04-04T21:39:08.368340+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:09:54.791797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2288,"reported_epoch":1194,"state":"active+clean","last_fresh":"2024-04-04T21:39:08.424813+0000","last_change":"2024-04-04T21:03:47.554482+0000","last_active":"2024-04-04T21:39:08.424813+0000","last_peered":"2024-04-04T21:39:08.424813+0000","last_clean":"2024-04-04T21:39:08.424813+0000","last_became_active":"2024-04-04T21:03:47.524251+0000","last_became_peered":"2024-04-04T21:03:47.524251+0000","last_unstale":"2024-04-04T21:39:08.424813+0000","last_undegraded":"2024-04-04T21:39:08.424813+0000","last_fullsized":"2024-04-04T21:39:08.424813+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:11:26.408285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2308,"reported_epoch":1195,"state":"active+clean","last_fresh":"2024-04-04T21:39:09.252850+0000","last_change":"2024-04-04T21:03:45.814604+0000","last_active":"2024-04-04T21:39:09.252850+0000","last_peered":"2024-04-04T21:39:09.252850+0000","last_clean":"2024-04-04T21:39:09.252850+0000","last_became_active":"2024-04-04T21:03:45.808883+0000","last_became_peered":"2024-04-04T21:03:45.808883+0000","last_unstale":"2024-04-04T21:39:09.252850+0000","last_undegraded":"2024-04-04T21:39:09.252850+0000","last_fullsized":"2024-04-04T21:39:09.252850+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:56:32.831438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899346367,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68324,"kb_used_data":5204,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303516,"statfs":{"total":96636764160,"available":96566800384,"internally_reserved":0,"allocated":5328896,"data_stored":2606125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346367,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72484,"kb_used_data":5204,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299356,"statfs":{"total":96636764160,"available":96562540544,"internally_reserved":0,"allocated":5328896,"data_stored":2606125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346367,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72856,"kb_used_data":5580,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298984,"statfs":{"total":96636764160,"available":96562159616,"internally_reserved":0,"allocated":5713920,"data_stored":2996281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411777,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72484,"kb_used_data":5208,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299356,"statfs":{"total":96636764160,"available":96562540544,"internally_reserved":0,"allocated":5332992,"data_stored":2606130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509889,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72468,"kb_used_data":5188,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299372,"statfs":{"total":96636764160,"available":96562556928,"internally_reserved":0,"allocated":5312512,"data_stored":2598441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509884,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68300,"kb_used_data":5176,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303540,"statfs":{"total":96636764160,"available":96566824960,"internally_reserved":0,"allocated":5300224,"data_stored":2590993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542591,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72500,"kb_used_data":5220,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299340,"statfs":{"total":96636764160,"available":96562524160,"internally_reserved":0,"allocated":5345280,"data_stored":2612837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575293,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68712,"kb_used_data":5596,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303128,"statfs":{"total":96636764160,"available":96566403072,"internally_reserved":0,"allocated":5730304,"data_stored":3003965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-04T21:39:12.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-04T21:39:12.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.901+0000 7fa8bef8a640 10 monclient: get_monmap_and_config 2024-04-04T21:39:12.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.901+0000 7fa8bef8a640 10 monclient: build_initial_monmap 2024-04-04T21:39:12.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.901+0000 7fa8bef8a640 10 monclient: monmap: 2024-04-04T21:39:12.902 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:12.902 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:12.902 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:12.902728+0000 2024-04-04T21:39:12.902 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:12.902728+0000 2024-04-04T21:39:12.902 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:12.902 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:12.902 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:12.902 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:12.902 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:12.902 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:12.902 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:12.902 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:12.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.902+0000 7fa8bef8a640 1 Processor -- start 2024-04-04T21:39:12.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.902+0000 7fa8bef8a640 1 -- start start 2024-04-04T21:39:12.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.902+0000 7fa8bef8a640 10 monclient: init 2024-04-04T21:39:12.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.902+0000 7fa8bef8a640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:12.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.902+0000 7fa8bef8a640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-04T21:39:12.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.902+0000 7fa8bef8a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa8b80b08a0 0x7fa8b8143bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:12.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.902+0000 7fa8bef8a640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b80b08a0 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:12.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.903+0000 7fa8bef8a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8b80b0c80 addr v1:172.21.15.203:6789/0 2024-04-04T21:39:12.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.903+0000 7fa8bef8a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8b80b14b0 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:12.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.903+0000 7fa8bef8a640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b81440f0 con 0x7fa8b80b14b0 2024-04-04T21:39:12.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.903+0000 7fa8bef8a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b8071c30 con 0x7fa8b80b0c80 2024-04-04T21:39:12.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.903+0000 7fa8bef8a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:12.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.903+0000 7fa8bef8a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b80718f0 con 0x7fa8b80b08a0 2024-04-04T21:39:12.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.903+0000 7fa8bef8a640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:12.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.903+0000 7fa8bef8a640 10 monclient(hunting): authenticate will time out at 3602.170240s 2024-04-04T21:39:12.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.903+0000 7fa8bd527640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fa8b80b14b0 0x7fa8b813dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:55150/0 (socket says 172.21.15.106:55150) 2024-04-04T21:39:12.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.903+0000 7fa8bd527640 1 -- 172.21.15.106:0/4189989804 learned_addr learned my addr 172.21.15.106:0/4189989804 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:39:12.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.903+0000 7fa8bcd26640 1 --2- 172.21.15.106:0/4189989804 >> v2:172.21.15.203:3300/0 conn(0x7fa8b80b08a0 0x7fa8b8143bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:12.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.904+0000 7fa8bcd26640 10 monclient(hunting): get_auth_request con 0x7fa8b80b08a0 auth_method 0 2024-04-04T21:39:12.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.904+0000 7fa8bcd26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:12.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.904+0000 7fa8bcd26640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:12.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.904+0000 7fa8bcd26640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:12.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.904+0000 7fa8bcd26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:12.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.904+0000 7fa8bcd26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:12.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.905+0000 7fa8bcd26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:12.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.906+0000 7fa8af7fe640 1 -- 172.21.15.106:0/4189989804 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883308127 0 0) 0x7fa8b81440f0 con 0x7fa8b80b14b0 2024-04-04T21:39:12.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.906+0000 7fa8af7fe640 10 monclient(hunting): _negotiate 2024-04-04T21:39:12.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.906+0000 7fa8af7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:12.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.906+0000 7fa8af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:12.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.906+0000 7fa8af7fe640 10 monclient(hunting): my global_id is 15099 2024-04-04T21:39:12.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.906+0000 7fa8af7fe640 1 -- 172.21.15.106:0/4189989804 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a0003520 con 0x7fa8b80b14b0 2024-04-04T21:39:12.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.907+0000 7fa8bcd26640 10 monclient(hunting): handle_auth_done global_id 22039 payload 274 2024-04-04T21:39:12.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.907+0000 7fa8bcd26640 1 -- 172.21.15.106:0/4189989804 >> v1:172.21.15.203:6789/0 conn(0x7fa8b80b0c80 legacy=0x7fa8b80aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:12.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.907+0000 7fa8bcd26640 1 -- 172.21.15.106:0/4189989804 >> v1:172.21.15.106:6789/0 conn(0x7fa8b80b14b0 legacy=0x7fa8b813dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:12.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.907+0000 7fa8bcd26640 10 monclient: _finish_hunting 0 2024-04-04T21:39:12.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.907+0000 7fa8bcd26640 1 monclient: found mon.noname-c 2024-04-04T21:39:12.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.907+0000 7fa8bcd26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:12.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.907+0000 7fa8bcd26640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:39:12.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.907+0000 7fa8bcd26640 1 -- 172.21.15.106:0/4189989804 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b8071e70 con 0x7fa8b80b08a0 2024-04-04T21:39:12.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.911+0000 7fa8bcd26640 1 --2- 172.21.15.106:0/4189989804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8b80b08a0 0x7fa8b8143bb0 secure :-1 s=READY pgs=5531 cs=0 l=1 rev1=1 crypto rx=0x7fa8b40097a0 tx=0x7fa8b402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ee70cd7ea5e08da server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:12.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8af7fe640 1 -- 172.21.15.106:0/4189989804 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4035020 con 0x7fa8b80b08a0 2024-04-04T21:39:12.915 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8af7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8af7fe640 10 monclient: dump: 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8af7fe640 10 monclient: _finish_auth 0 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8af7fe640 10 monclient: _check_auth_tickets 2024-04-04T21:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8af7fe640 1 -- 172.21.15.106:0/4189989804 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b4031830 con 0x7fa8b80b08a0 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8af7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8af7fe640 1 -- 172.21.15.106:0/4189989804 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4031c00 con 0x7fa8b80b08a0 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8af7fe640 10 monclient: dump: 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:12.918 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:12.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8bef8a640 5 monclient: authenticate success, global_id 22039 2024-04-04T21:39:12.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8bef8a640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:12.918 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8bef8a640 4 set_mon_vals no callback set 2024-04-04T21:39:12.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8bef8a640 10 monclient: shutdown 2024-04-04T21:39:12.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8bef8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:12.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8bef8a640 1 -- 172.21.15.106:0/4189989804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8b80b08a0 msgr2=0x7fa8b8143bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:12.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8bef8a640 1 --2- 172.21.15.106:0/4189989804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8b80b08a0 0x7fa8b8143bb0 secure :-1 s=READY pgs=5531 cs=0 l=1 rev1=1 crypto rx=0x7fa8b40097a0 tx=0x7fa8b402b8d0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8bef8a640 1 -- 172.21.15.106:0/4189989804 shutdown_connections 2024-04-04T21:39:12.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8bef8a640 1 --2- 172.21.15.106:0/4189989804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8b80b08a0 0x7fa8b8143bb0 unknown :-1 s=CLOSED pgs=5531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8bef8a640 1 -- 172.21.15.106:0/4189989804 >> 172.21.15.106:0/4189989804 conn(0x7fa8b80b1990 msgr2=0x7fa8b80b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:12.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8bef8a640 1 -- 172.21.15.106:0/4189989804 shutdown_connections 2024-04-04T21:39:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.916+0000 7fa8bef8a640 1 -- 172.21.15.106:0/4189989804 wait complete. 2024-04-04T21:39:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.917+0000 7fa8bef8a640 10 monclient: build_initial_monmap 2024-04-04T21:39:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.917+0000 7fa8bef8a640 10 monclient: monmap: 2024-04-04T21:39:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:12.922 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.917+0000 7fa8bef8a640 1 librados: starting msgr at 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.917+0000 7fa8bef8a640 1 librados: starting objecter 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.917+0000 7fa8bef8a640 1 Processor -- start 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.917+0000 7fa8bef8a640 1 -- start start 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.917+0000 7fa8bef8a640 1 librados: setting wanted keys 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.917+0000 7fa8bef8a640 1 librados: calling monclient init 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.917+0000 7fa8bef8a640 10 monclient: init 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8b80b08a0 0x7fa8b81d6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b80b08a0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8b80b14b0 0x7fa8b813bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 10 monclient(hunting): picked mon.noname-b con 0x7fa8b80b14b0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8b813c180 0x7fa8b8142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b813c180 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b806f640 con 0x7fa8b80b08a0 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b806f4c0 con 0x7fa8b80b14b0 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b806f7c0 con 0x7fa8b813c180 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bef8a640 10 monclient(hunting): authenticate will time out at 3602.185283s 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bcd26640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8b80b08a0 0x7fa8b81d6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8affff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8b80b14b0 0x7fa8b813bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.918+0000 7fa8bd527640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8b813c180 0x7fa8b8142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bcd26640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8b80b08a0 0x7fa8b81d6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:60924/0 (socket says 172.21.15.106:60924) 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bcd26640 1 -- 172.21.15.106:0/1194187458 learned_addr learned my addr 172.21.15.106:0/1194187458 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bcd26640 10 monclient(hunting): get_auth_request con 0x7fa8b80b08a0 auth_method 0 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bcd26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:12.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bcd26640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bcd26640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8affff640 10 monclient(hunting): get_auth_request con 0x7fa8b80b14b0 auth_method 0 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8affff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bd527640 10 monclient(hunting): get_auth_request con 0x7fa8b813c180 auth_method 0 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bd527640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bd527640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bd527640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bcd26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bcd26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bcd26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bd527640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:12.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bd527640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:12.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.919+0000 7fa8bd527640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:12.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.922+0000 7fa8affff640 10 monclient(hunting): handle_auth_done global_id 22045 payload 824 2024-04-04T21:39:12.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.922+0000 7fa8affff640 1 -- 172.21.15.106:0/1194187458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8b813c180 msgr2=0x7fa8b8142220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:12.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.922+0000 7fa8affff640 1 --2- 172.21.15.106:0/1194187458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8b813c180 0x7fa8b8142220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.922+0000 7fa8affff640 1 -- 172.21.15.106:0/1194187458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8b80b08a0 msgr2=0x7fa8b81d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:12.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.922+0000 7fa8affff640 1 --2- 172.21.15.106:0/1194187458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8b80b08a0 0x7fa8b81d6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:12.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.922+0000 7fa8affff640 10 monclient: _finish_hunting 0 2024-04-04T21:39:12.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.922+0000 7fa8affff640 1 monclient: found mon.noname-b 2024-04-04T21:39:12.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.922+0000 7fa8affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:12.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.922+0000 7fa8affff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:12.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.922+0000 7fa8affff640 1 -- 172.21.15.106:0/1194187458 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b8141d00 con 0x7fa8b80b14b0 2024-04-04T21:39:12.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.924+0000 7fa8affff640 1 --2- 172.21.15.106:0/1194187458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8b80b14b0 0x7fa8b813bc40 secure :-1 s=READY pgs=5532 cs=0 l=1 rev1=1 crypto rx=0x7fa8a800c340 tx=0x7fa8a800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:12.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.924+0000 7fa8ad7fa640 10 client.?.objecter ms_handle_connect 0x7fa8b80b14b0 2024-04-04T21:39:12.927 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.924+0000 7fa8ad7fa640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:12.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8ad7fa640 1 -- 172.21.15.106:0/1194187458 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a80052d0 con 0x7fa8b80b14b0 2024-04-04T21:39:12.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:12.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8ad7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:12.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8ad7fa640 10 monclient: dump: 2024-04-04T21:39:12.929 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:12.929 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:12.929 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.929 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.929 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:12.929 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:12.929 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:12.929 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:12.929 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:12.930 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:12.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8ad7fa640 10 monclient: _finish_auth 0 2024-04-04T21:39:12.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8ad7fa640 10 monclient: _check_auth_tickets 2024-04-04T21:39:12.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:12.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8ad7fa640 1 -- 172.21.15.106:0/1194187458 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a8005490 con 0x7fa8b80b14b0 2024-04-04T21:39:12.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8bef8a640 5 monclient: authenticate success, global_id 22045 2024-04-04T21:39:12.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8bef8a640 10 monclient: _renew_subs 2024-04-04T21:39:12.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8bef8a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:12.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8bef8a640 1 -- 172.21.15.106:0/1194187458 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b81d1110 con 0x7fa8b80b14b0 2024-04-04T21:39:12.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:12.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8ad7fa640 1 -- 172.21.15.106:0/1194187458 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8007260 con 0x7fa8b80b14b0 2024-04-04T21:39:12.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:12.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8ad7fa640 10 monclient: dump: 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8aeffd640 4 set_mon_vals no callback set 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8bef8a640 10 client.22045.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8bef8a640 10 monclient: _renew_subs 2024-04-04T21:39:12.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8bef8a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:12.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8bef8a640 1 -- 172.21.15.106:0/1194187458 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b813c600 con 0x7fa8b80b14b0 2024-04-04T21:39:12.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8bef8a640 1 librados: init done 2024-04-04T21:39:12.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8bef8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:12.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8bef8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:12.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8bef8a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:12.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:12.928+0000 7fa8bef8a640 1 -- 172.21.15.106:0/1194187458 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b804f050 con 0x7fa8b80b14b0 2024-04-04T21:39:13.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.177+0000 7fa8ad7fa640 1 -- 172.21.15.106:0/1194187458 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa8a8007400 con 0x7fa8b80b14b0 2024-04-04T21:39:13.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.178+0000 7fa8ad7fa640 1 --2- 172.21.15.106:0/1194187458 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa8a406ceb0 0x7fa8a406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:13.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.178+0000 7fa8bcd26640 1 --2- 172.21.15.106:0/1194187458 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa8a406ceb0 0x7fa8a406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:13.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.178+0000 7fa8bcd26640 10 monclient: get_auth_request con 0x7fa8a406ceb0 auth_method 0 2024-04-04T21:39:13.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.178+0000 7fa8ad7fa640 10 client.22045.objecter ms_handle_connect 0x7fa8a406ceb0 2024-04-04T21:39:13.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.178+0000 7fa8bcd26640 1 --2- 172.21.15.106:0/1194187458 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa8a406ceb0 0x7fa8a406f370 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009560 tx=0x7fa8b4008030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:13.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.180+0000 7fa8ad7fa640 1 -- 172.21.15.106:0/1194187458 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7fa8a80563a0 con 0x7fa8b80b14b0 2024-04-04T21:39:13.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.180+0000 7fa8ad7fa640 10 client.22045.objecter ms_dispatch 0x7fa8b8000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:13.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.180+0000 7fa8ad7fa640 3 client.22045.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:13.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.180+0000 7fa8ad7fa640 3 client.22045.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:13.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.180+0000 7fa8ad7fa640 20 client.22045.objecter dump_active .. 0 homeless 2024-04-04T21:39:13.190 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.190+0000 7fa8ad7fa640 1 -- 172.21.15.106:0/1194187458 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa8a8058300 con 0x7fa8b80b14b0 2024-04-04T21:39:13.190 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.190+0000 7fa8ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:13.190 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.190+0000 7fa8ad7fa640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:13.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.362+0000 7fa8bef8a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:39:13.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.362+0000 7fa8bef8a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:39:13.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.362+0000 7fa8bef8a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:13.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.362+0000 7fa8bef8a640 1 -- 172.21.15.106:0/1194187458 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8b8141ed0 con 0x7fa8b80b14b0 2024-04-04T21:39:13.399 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.398+0000 7fa8ad7fa640 1 -- 172.21.15.106:0/1194187458 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1197) v1 ==== 74+0+17770 (secure 0 0 0) 0x7fa8a805c3a0 con 0x7fa8b80b14b0 2024-04-04T21:39:13.399 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.398+0000 7fa8ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-04T21:39:13.399 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.398+0000 7fa8ad7fa640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:39:13.399 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:39:13.400 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1197,"fsid":"8f89ce43-1bcb-41cf-9555-c40eeb258155","created":"2024-04-04T21:01:20.057083+0000","modified":"2024-04-04T21:39:10.866724+0000","last_up_change":"2024-04-04T21:02:16.990959+0000","last_in_change":"2024-04-04T21:01:25.642687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":116,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-04T21:02:04.746208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-04T21:02:21.817409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-04-04T21:03:34.214509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"28337134-5253-454a-bcda-47267b05b2c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6819","nonce":2744620362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6821","nonce":2744620362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6825","nonce":2744620362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":2744620362},{"type":"v1","addr":"172.21.15.106:6823","nonce":2744620362}]},"public_addr":"172.21.15.106:6819/2744620362","cluster_addr":"172.21.15.106:6821/2744620362","heartbeat_back_addr":"172.21.15.106:6825/2744620362","heartbeat_front_addr":"172.21.15.106:6823/2744620362","state":["exists","up"]},{"osd":1,"uuid":"ae336a0e-2a90-4323-9f0f-1701fffe6b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6827","nonce":2321665741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6829","nonce":2321665741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6833","nonce":2321665741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":2321665741},{"type":"v1","addr":"172.21.15.106:6831","nonce":2321665741}]},"public_addr":"172.21.15.106:6827/2321665741","cluster_addr":"172.21.15.106:6829/2321665741","heartbeat_back_addr":"172.21.15.106:6833/2321665741","heartbeat_front_addr":"172.21.15.106:6831/2321665741","state":["exists","up"]},{"osd":2,"uuid":"773087e0-78c9-4d14-bf0a-fb605c21f92f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6811","nonce":459340769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6813","nonce":459340769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6817","nonce":459340769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":459340769},{"type":"v1","addr":"172.21.15.106:6815","nonce":459340769}]},"public_addr":"172.21.15.106:6811/459340769","cluster_addr":"172.21.15.106:6813/459340769","heartbeat_back_addr":"172.21.15.106:6817/459340769","heartbeat_front_addr":"172.21.15.106:6815/459340769","state":["exists","up"]},{"osd":3,"uuid":"07da9513-c746-46d1-995f-6f8e91338b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6803","nonce":873500775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6805","nonce":873500775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6809","nonce":873500775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":873500775},{"type":"v1","addr":"172.21.15.106:6807","nonce":873500775}]},"public_addr":"172.21.15.106:6803/873500775","cluster_addr":"172.21.15.106:6805/873500775","heartbeat_back_addr":"172.21.15.106:6809/873500775","heartbeat_front_addr":"172.21.15.106:6807/873500775","state":["exists","up"]},{"osd":4,"uuid":"c47d034c-1db6-4332-bf24-722fb2846808","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6825","nonce":2225997929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6827","nonce":2225997929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6831","nonce":2225997929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2225997929},{"type":"v1","addr":"172.21.15.203:6829","nonce":2225997929}]},"public_addr":"172.21.15.203:6825/2225997929","cluster_addr":"172.21.15.203:6827/2225997929","heartbeat_back_addr":"172.21.15.203:6831/2225997929","heartbeat_front_addr":"172.21.15.203:6829/2225997929","state":["exists","up"]},{"osd":5,"uuid":"588caa62-e773-44ae-94e5-ece54fe980d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6801","nonce":696438187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6803","nonce":696438187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6807","nonce":696438187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":696438187},{"type":"v1","addr":"172.21.15.203:6805","nonce":696438187}]},"public_addr":"172.21.15.203:6801/696438187","cluster_addr":"172.21.15.203:6803/696438187","heartbeat_back_addr":"172.21.15.203:6807/696438187","heartbeat_front_addr":"172.21.15.203:6805/696438187","state":["exists","up"]},{"osd":6,"uuid":"ef776e12-7c80-4095-b173-5b2d24996f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6817","nonce":2758852830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6819","nonce":2758852830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6823","nonce":2758852830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2758852830},{"type":"v1","addr":"172.21.15.203:6821","nonce":2758852830}]},"public_addr":"172.21.15.203:6817/2758852830","cluster_addr":"172.21.15.203:6819/2758852830","heartbeat_back_addr":"172.21.15.203:6823/2758852830","heartbeat_front_addr":"172.21.15.203:6821/2758852830","state":["exists","up"]},{"osd":7,"uuid":"53aafc58-6463-4aeb-8395-e14fcfcee5de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6809","nonce":4104640840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6811","nonce":4104640840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6815","nonce":4104640840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4104640840},{"type":"v1","addr":"172.21.15.203:6813","nonce":4104640840}]},"public_addr":"172.21.15.203:6809/4104640840","cluster_addr":"172.21.15.203:6811/4104640840","heartbeat_back_addr":"172.21.15.203:6815/4104640840","heartbeat_front_addr":"172.21.15.203:6813/4104640840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:03.982282+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:05.073358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:04.386488+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:04.994186+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:02.522349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:01.591956+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:02.315592+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-04T21:02:01.851115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":10,"snaps":[{"begin":5,"length":2}]},{"pool":13,"snaps":[{"begin":3,"length":2}]},{"pool":19,"snaps":[{"begin":1,"length":1}]},{"pool":70,"snaps":[{"begin":1,"length":1}]},{"pool":101,"snaps":[{"begin":7,"length":2}]},{"pool":103,"snaps":[{"begin":1,"length":1}]},{"pool":109,"snaps":[{"begin":1,"length":2}]},{"pool":119,"snaps":[{"begin":3,"length":1}]},{"pool":123,"snaps":[{"begin":3,"length":2}]},{"pool":125,"snaps":[{"begin":2,"length":2}]},{"pool":129,"snaps":[{"begin":2,"length":3}]},{"pool":132,"snaps":[{"begin":4,"length":2}]},{"pool":136,"snaps":[{"begin":2,"length":2}]},{"pool":139,"snaps":[{"begin":2,"length":1}]},{"pool":141,"snaps":[{"begin":1,"length":4}]},{"pool":143,"snaps":[{"begin":3,"length":2},{"begin":6,"length":1}]},{"pool":144,"snaps":[{"begin":4,"length":4}]},{"pool":162,"snaps":[{"begin":7,"length":1}]},{"pool":166,"snaps":[{"begin":8,"length":1}]},{"pool":168,"snaps":[{"begin":10,"length":1}]},{"pool":169,"snaps":[{"begin":11,"length":2}]},{"pool":172,"snaps":[{"begin":14,"length":1}]},{"pool":217,"snaps":[{"begin":2,"length":1}]},{"pool":218,"snaps":[{"begin":3,"length":1}]},{"pool":221,"snaps":[{"begin":4,"length":1}]},{"pool":225,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-04T21:39:13.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.400+0000 7fa8bef8a640 10 librados: watch_flush enter 2024-04-04T21:39:13.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.400+0000 7fa8bef8a640 10 librados: watch_flush exit 2024-04-04T21:39:13.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.400+0000 7fa8bef8a640 20 client.22045.objecter shutdown clearing up homeless session... 2024-04-04T21:39:13.401 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.400+0000 7fa8bef8a640 10 client.22045.objecter successfully canceled tick 2024-04-04T21:39:13.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.400+0000 7fa8bef8a640 1 -- 172.21.15.106:0/1194187458 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa8a406ceb0 msgr2=0x7fa8a406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:13.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.400+0000 7fa8bef8a640 1 --2- 172.21.15.106:0/1194187458 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa8a406ceb0 0x7fa8a406f370 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009560 tx=0x7fa8b4008030 comp rx=0 tx=0).stop 2024-04-04T21:39:13.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.400+0000 7fa8bef8a640 10 monclient: shutdown 2024-04-04T21:39:13.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.400+0000 7fa8bef8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:13.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.400+0000 7fa8bef8a640 1 -- 172.21.15.106:0/1194187458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8b80b14b0 msgr2=0x7fa8b813bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:13.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.400+0000 7fa8bef8a640 1 --2- 172.21.15.106:0/1194187458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8b80b14b0 0x7fa8b813bc40 secure :-1 s=READY pgs=5532 cs=0 l=1 rev1=1 crypto rx=0x7fa8a800c340 tx=0x7fa8a800c800 comp rx=0 tx=0).stop 2024-04-04T21:39:13.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.401+0000 7fa8bef8a640 1 -- 172.21.15.106:0/1194187458 shutdown_connections 2024-04-04T21:39:13.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.401+0000 7fa8bef8a640 1 --2- 172.21.15.106:0/1194187458 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa8a406ceb0 0x7fa8a406f370 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:13.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.401+0000 7fa8bef8a640 1 --2- 172.21.15.106:0/1194187458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8b813c180 0x7fa8b8142220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:13.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.401+0000 7fa8bef8a640 1 --2- 172.21.15.106:0/1194187458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8b80b14b0 0x7fa8b813bc40 unknown :-1 s=CLOSED pgs=5532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:13.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.401+0000 7fa8bef8a640 1 --2- 172.21.15.106:0/1194187458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8b80b08a0 0x7fa8b81d6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:13.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.401+0000 7fa8bef8a640 1 -- 172.21.15.106:0/1194187458 >> 172.21.15.106:0/1194187458 conn(0x7fa8b80b1990 msgr2=0x7fa8b80b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:13.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.401+0000 7fa8bef8a640 1 -- 172.21.15.106:0/1194187458 shutdown_connections 2024-04-04T21:39:13.402 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.401+0000 7fa8bef8a640 1 -- 172.21.15.106:0/1194187458 wait complete. 2024-04-04T21:39:13.403 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:13.401+0000 7fa8bef8a640 1 librados: shutdown 2024-04-04T21:39:14.414 INFO:tasks.ceph:Scrubbing osd.0 2024-04-04T21:39:14.414 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-04T21:39:14.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.525+0000 7fa7a8047640 10 monclient: get_monmap_and_config 2024-04-04T21:39:14.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.525+0000 7fa7a8047640 10 monclient: build_initial_monmap 2024-04-04T21:39:14.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.525+0000 7fa7a8047640 10 monclient: monmap: 2024-04-04T21:39:14.525 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:14.525 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:14.525 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:14.525605+0000 2024-04-04T21:39:14.525 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:14.525605+0000 2024-04-04T21:39:14.525 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:14.525 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:14.525 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:14.525 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:14.525 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:14.525 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:14.525 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:14.525 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:14.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 1 Processor -- start 2024-04-04T21:39:14.527 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 1 -- start start 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 10 monclient: init 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7a00b1c00 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7a013d780 addr v1:172.21.15.203:6789/0 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa7a0140540 0x7fa7a01409b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a0140540 addr v2:172.21.15.106:3300/0 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a00a9670 con 0x7fa7a00b1c00 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a0144150 con 0x7fa7a013d780 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a00b1fe0 con 0x7fa7a0140540 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.526+0000 7fa7a8047640 10 monclient(hunting): authenticate will time out at 3603.793103s 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.527+0000 7fa7a65e4640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa7a0140540 0x7fa7a01409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:14.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.527+0000 7fa7a55e2640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa7a013d780 0x7fa7a013dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:55636/0 (socket says 172.21.15.106:55636) 2024-04-04T21:39:14.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.527+0000 7fa7a55e2640 1 -- 172.21.15.106:0/2874279282 learned_addr learned my addr 172.21.15.106:0/2874279282 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:39:14.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.528+0000 7fa7a65e4640 10 monclient(hunting): get_auth_request con 0x7fa7a0140540 auth_method 0 2024-04-04T21:39:14.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.528+0000 7fa7a65e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:14.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.528+0000 7fa7a65e4640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:14.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.528+0000 7fa7a65e4640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:14.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.528+0000 7fa7a65e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:14.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.528+0000 7fa7a65e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:14.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.528+0000 7fa7a65e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:14.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.530+0000 7fa7a65e4640 10 monclient(hunting): handle_auth_done global_id 15105 payload 274 2024-04-04T21:39:14.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.530+0000 7fa7a65e4640 1 -- 172.21.15.106:0/2874279282 >> v1:172.21.15.203:6789/0 conn(0x7fa7a013d780 legacy=0x7fa7a013dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:14.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.530+0000 7fa7a65e4640 1 -- 172.21.15.106:0/2874279282 >> v1:172.21.15.106:6789/0 conn(0x7fa7a00b1c00 legacy=0x7fa7a0143a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:14.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.530+0000 7fa7a65e4640 10 monclient: _finish_hunting 0 2024-04-04T21:39:14.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.530+0000 7fa7a65e4640 1 monclient: found mon.noname-a 2024-04-04T21:39:14.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.530+0000 7fa7a65e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:14.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.530+0000 7fa7a65e4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:39:14.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.530+0000 7fa7a65e4640 1 -- 172.21.15.106:0/2874279282 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a0140ef0 con 0x7fa7a0140540 2024-04-04T21:39:14.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.532+0000 7fa7a65e4640 1 --2- 172.21.15.106:0/2874279282 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7a0140540 0x7fa7a01409b0 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 crypto rx=0x7fa798009d90 tx=0x7fa79802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f33891f0447913 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:14.536 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a4de1640 1 -- 172.21.15.106:0/2874279282 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa798039070 con 0x7fa7a0140540 2024-04-04T21:39:14.536 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a4de1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:14.536 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a4de1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:39:14.536 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a4de1640 10 monclient: dump: 2024-04-04T21:39:14.536 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:14.536 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:14.536 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.536 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.536 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a4de1640 10 monclient: _finish_auth 0 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a4de1640 10 monclient: _check_auth_tickets 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a4de1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a4de1640 1 -- 172.21.15.106:0/2874279282 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa79802cb40 con 0x7fa7a0140540 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a4de1640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a4de1640 1 -- 172.21.15.106:0/2874279282 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa798034440 con 0x7fa7a0140540 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a8047640 5 monclient: authenticate success, global_id 15105 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a8047640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a8047640 4 set_mon_vals no callback set 2024-04-04T21:39:14.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a4de1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:14.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a4de1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:39:14.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a4de1640 10 monclient: dump: 2024-04-04T21:39:14.538 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:14.538 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:14.538 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.538 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.538 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:14.538 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:14.538 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:14.538 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:14.538 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:14.538 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:14.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a8047640 10 monclient: shutdown 2024-04-04T21:39:14.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a8047640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:14.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a8047640 1 -- 172.21.15.106:0/2874279282 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7a0140540 msgr2=0x7fa7a01409b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:14.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a8047640 1 --2- 172.21.15.106:0/2874279282 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7a0140540 0x7fa7a01409b0 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 crypto rx=0x7fa798009d90 tx=0x7fa79802bd90 comp rx=0 tx=0).stop 2024-04-04T21:39:14.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a8047640 1 -- 172.21.15.106:0/2874279282 shutdown_connections 2024-04-04T21:39:14.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a8047640 1 --2- 172.21.15.106:0/2874279282 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7a0140540 0x7fa7a01409b0 unknown :-1 s=CLOSED pgs=3159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:14.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.537+0000 7fa7a8047640 1 -- 172.21.15.106:0/2874279282 >> 172.21.15.106:0/2874279282 conn(0x7fa7a00b2a80 msgr2=0x7fa7a00b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:14.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.538+0000 7fa7a8047640 1 -- 172.21.15.106:0/2874279282 shutdown_connections 2024-04-04T21:39:14.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.538+0000 7fa7a8047640 1 -- 172.21.15.106:0/2874279282 wait complete. 2024-04-04T21:39:14.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.538+0000 7fa7a8047640 10 monclient: build_initial_monmap 2024-04-04T21:39:14.542 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.538+0000 7fa7a8047640 10 monclient: monmap: 2024-04-04T21:39:14.542 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:14.542 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 1 librados: starting msgr at 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 1 librados: starting objecter 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 1 Processor -- start 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 1 -- start start 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 1 librados: setting wanted keys 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 1 librados: calling monclient init 2024-04-04T21:39:14.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 10 monclient: init 2024-04-04T21:39:14.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:14.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:39:14.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7a00b1c00 0x7fa7a006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:14.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 10 monclient(hunting): picked mon.noname-c con 0x7fa7a00b1c00 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:14.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7a013d780 0x7fa7a013c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:14.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a013d780 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:14.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7a0140540 0x7fa7a013cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:14.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a0140540 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:14.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:14.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a0144970 con 0x7fa7a0140540 2024-04-04T21:39:14.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:14.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a01444e0 con 0x7fa7a013d780 2024-04-04T21:39:14.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:14.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a014bfb0 con 0x7fa7a00b1c00 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.539+0000 7fa7a8047640 10 monclient(hunting): authenticate will time out at 3603.806093s 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a5de3640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7a00b1c00 0x7fa7a006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a65e4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7a0140540 0x7fa7a013cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a55e2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7a013d780 0x7fa7a013c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a5de3640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7a00b1c00 0x7fa7a006d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:38068/0 (socket says 172.21.15.106:38068) 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a5de3640 1 -- 172.21.15.106:0/2639185336 learned_addr learned my addr 172.21.15.106:0/2639185336 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a65e4640 10 monclient(hunting): get_auth_request con 0x7fa7a0140540 auth_method 0 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a65e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a65e4640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a65e4640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a5de3640 10 monclient(hunting): get_auth_request con 0x7fa7a00b1c00 auth_method 0 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a5de3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a5de3640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:14.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a5de3640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:14.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a55e2640 10 monclient(hunting): get_auth_request con 0x7fa7a013d780 auth_method 0 2024-04-04T21:39:14.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a55e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:14.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a55e2640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:14.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.540+0000 7fa7a55e2640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:14.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.541+0000 7fa7a65e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:14.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.541+0000 7fa7a65e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:14.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.541+0000 7fa7a65e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:14.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.541+0000 7fa7a5de3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:14.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.541+0000 7fa7a5de3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:14.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.541+0000 7fa7a5de3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:14.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.541+0000 7fa7a55e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:14.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.541+0000 7fa7a55e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:14.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.541+0000 7fa7a55e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:14.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.544+0000 7fa7a5de3640 10 monclient(hunting): handle_auth_done global_id 17642 payload 824 2024-04-04T21:39:14.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.544+0000 7fa7a5de3640 1 -- 172.21.15.106:0/2639185336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7a013d780 msgr2=0x7fa7a013c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:14.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.544+0000 7fa7a5de3640 1 --2- 172.21.15.106:0/2639185336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7a013d780 0x7fa7a013c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:14.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.544+0000 7fa7a5de3640 1 -- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7a0140540 msgr2=0x7fa7a013cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:14.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.544+0000 7fa7a5de3640 1 --2- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7a0140540 0x7fa7a013cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:14.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.544+0000 7fa7a5de3640 10 monclient: _finish_hunting 0 2024-04-04T21:39:14.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.544+0000 7fa7a5de3640 1 monclient: found mon.noname-c 2024-04-04T21:39:14.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.544+0000 7fa7a5de3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:14.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.544+0000 7fa7a5de3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:39:14.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.544+0000 7fa7a5de3640 1 -- 172.21.15.106:0/2639185336 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a006d190 con 0x7fa7a00b1c00 2024-04-04T21:39:14.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.546+0000 7fa7a5de3640 1 --2- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7a00b1c00 0x7fa7a006d6b0 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7fa79c00c410 tx=0x7fa79c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:14.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.546+0000 7fa7967fc640 10 client.?.objecter ms_handle_connect 0x7fa7a00b1c00 2024-04-04T21:39:14.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.546+0000 7fa7967fc640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:14.549 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7967fc640 1 -- 172.21.15.106:0/2639185336 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c017070 con 0x7fa7a00b1c00 2024-04-04T21:39:14.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:14.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:39:14.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7967fc640 10 monclient: dump: 2024-04-04T21:39:14.550 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:14.550 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:14.550 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.550 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.550 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:14.550 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:14.550 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:14.550 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:14.550 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:14.550 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7967fc640 10 monclient: _finish_auth 0 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7967fc640 10 monclient: _check_auth_tickets 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7967fc640 1 -- 172.21.15.106:0/2639185336 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa79c0035f0 con 0x7fa7a00b1c00 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7967fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7967fc640 1 -- 172.21.15.106:0/2639185336 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c003a20 con 0x7fa7a00b1c00 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7967fc640 10 monclient: dump: 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:14.551 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:14.552 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:14.552 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:14.552 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:14.552 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:14.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7a8047640 5 monclient: authenticate success, global_id 17642 2024-04-04T21:39:14.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7a8047640 10 monclient: _renew_subs 2024-04-04T21:39:14.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa797fff640 4 set_mon_vals no callback set 2024-04-04T21:39:14.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7a8047640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:39:14.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7a8047640 1 -- 172.21.15.106:0/2639185336 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a013d0f0 con 0x7fa7a00b1c00 2024-04-04T21:39:14.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7a8047640 10 client.17642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:14.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7a8047640 10 monclient: _renew_subs 2024-04-04T21:39:14.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7a8047640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:39:14.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7a8047640 1 -- 172.21.15.106:0/2639185336 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a006ddd0 con 0x7fa7a00b1c00 2024-04-04T21:39:14.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.549+0000 7fa7a8047640 1 librados: init done 2024-04-04T21:39:14.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.550+0000 7fa7a8047640 10 client.17642.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:14.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.550+0000 7fa7a8047640 15 client.17642.objecter _session_command_op_assign -1 1 2024-04-04T21:39:14.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.550+0000 7fa7a8047640 20 client.17642.objecter _get_session osd=-1 returning homeless 2024-04-04T21:39:14.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.550+0000 7fa7a8047640 10 client.17642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:14.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.550+0000 7fa7a8047640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:39:14.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.550+0000 7fa7a8047640 1 -- 172.21.15.106:0/2639185336 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa7a013be20 con 0x7fa7a00b1c00 2024-04-04T21:39:14.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.569+0000 7fa7967fc640 1 -- 172.21.15.106:0/2639185336 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa79c01a070 con 0x7fa7a00b1c00 2024-04-04T21:39:14.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.569+0000 7fa7967fc640 1 --2- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa78406ceb0 0x7fa78406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:14.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.570+0000 7fa7a55e2640 1 --2- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa78406ceb0 0x7fa78406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:14.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.570+0000 7fa7a55e2640 10 monclient: get_auth_request con 0x7fa78406ceb0 auth_method 0 2024-04-04T21:39:14.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.570+0000 7fa7a55e2640 1 --2- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa78406ceb0 0x7fa78406f370 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa7a013bad0 tx=0x7fa790007e80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:14.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.570+0000 7fa7967fc640 10 client.17642.objecter ms_handle_connect 0x7fa78406ceb0 2024-04-04T21:39:14.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 1 -- 172.21.15.106:0/2639185336 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7fa79c0565b0 con 0x7fa7a00b1c00 2024-04-04T21:39:14.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 10 client.17642.objecter ms_dispatch 0x7fa7a0000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:14.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 3 client.17642.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:14.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 3 client.17642.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:14.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 10 client.17642.objecter checking command 1 2024-04-04T21:39:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 1 --2- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:6818/2744620362,v1:172.21.15.106:6819/2744620362] conn(0x7fa784074bc0 0x7fa784077060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 20 client.17642.objecter _get_session s=0x7fa7840744f0 osd=0 3 2024-04-04T21:39:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 20 client.17642.objecter put_session s=0x7fa7840744f0 osd=0 3 2024-04-04T21:39:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 15 client.17642.objecter _session_command_op_remove -1 1 2024-04-04T21:39:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 20 client.17642.objecter _get_session s=0x7fa7840744f0 osd=0 3 2024-04-04T21:39:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 20 client.17642.objecter get_session s=0x7fa7840744f0 osd=0 3 2024-04-04T21:39:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 15 client.17642.objecter _session_command_op_assign 0 1 2024-04-04T21:39:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 20 client.17642.objecter put_session s=0x7fa7840744f0 osd=0 4 2024-04-04T21:39:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 10 client.17642.objecter _send_command 1 2024-04-04T21:39:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 1 -- 172.21.15.106:0/2639185336 --> [v2:172.21.15.106:6818/2744620362,v1:172.21.15.106:6819/2744620362] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa784077730 con 0x7fa784074bc0 2024-04-04T21:39:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.571+0000 7fa7967fc640 20 client.17642.objecter dump_active .. 0 homeless 2024-04-04T21:39:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.572+0000 7fa7a65e4640 1 --2- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:6818/2744620362,v1:172.21.15.106:6819/2744620362] conn(0x7fa784074bc0 0x7fa784077060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.572+0000 7fa7967fc640 1 -- 172.21.15.106:0/2639185336 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_get_version_reply(handle=1 version=1197) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa79c0547b0 con 0x7fa7a00b1c00 2024-04-04T21:39:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.572+0000 7fa7967fc640 10 monclient: handle_get_version_reply finishing 1 version 1197 2024-04-04T21:39:14.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.572+0000 7fa7a65e4640 10 monclient: get_auth_request con 0x7fa784074bc0 auth_method 0 2024-04-04T21:39:14.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.575+0000 7fa7a65e4640 1 --2- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:6818/2744620362,v1:172.21.15.106:6819/2744620362] conn(0x7fa784074bc0 0x7fa784077060 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:14.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.575+0000 7fa7967fc640 10 client.17642.objecter ms_handle_connect 0x7fa784074bc0 2024-04-04T21:39:14.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.598+0000 7fa7967fc640 1 -- 172.21.15.106:0/2639185336 <== osd.0 v2:172.21.15.106:6818/2744620362 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa784077730 con 0x7fa784074bc0 2024-04-04T21:39:14.597 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.598+0000 7fa7967fc640 10 client.17642.objecter ms_dispatch 0x7fa7a0000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:39:14.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.598+0000 7fa7967fc640 10 client.17642.objecter _finish_command 1 = system:0 2024-04-04T21:39:14.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.598+0000 7fa7967fc640 20 client.17642.objecter put_session s=0x7fa7840744f0 osd=0 4 2024-04-04T21:39:14.598 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.598+0000 7fa7967fc640 15 client.17642.objecter _session_command_op_remove 0 1 2024-04-04T21:39:14.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.611+0000 7fa7a8047640 10 client.17642.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-04T21:39:14.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.611+0000 7fa7a8047640 15 client.17642.objecter _session_command_op_assign -1 2 2024-04-04T21:39:14.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.611+0000 7fa7a8047640 20 client.17642.objecter _get_session s=0x7fa7840744f0 osd=0 3 2024-04-04T21:39:14.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.611+0000 7fa7a8047640 20 client.17642.objecter put_session s=0x7fa7840744f0 osd=0 3 2024-04-04T21:39:14.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.611+0000 7fa7a8047640 20 client.17642.objecter _get_session s=0x7fa7840744f0 osd=0 3 2024-04-04T21:39:14.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.611+0000 7fa7a8047640 15 client.17642.objecter _session_command_op_remove -1 2 2024-04-04T21:39:14.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.611+0000 7fa7a8047640 20 client.17642.objecter get_session s=0x7fa7840744f0 osd=0 3 2024-04-04T21:39:14.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.611+0000 7fa7a8047640 15 client.17642.objecter _session_command_op_assign 0 2 2024-04-04T21:39:14.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.611+0000 7fa7a8047640 20 client.17642.objecter put_session s=0x7fa7840744f0 osd=0 4 2024-04-04T21:39:14.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.611+0000 7fa7a8047640 10 client.17642.objecter _send_command 2 2024-04-04T21:39:14.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.611+0000 7fa7a8047640 1 -- 172.21.15.106:0/2639185336 --> [v2:172.21.15.106:6818/2744620362,v1:172.21.15.106:6819/2744620362] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa7a00685a0 con 0x7fa784074bc0 2024-04-04T21:39:14.624 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7967fc640 1 -- 172.21.15.106:0/2639185336 <== osd.0 v2:172.21.15.106:6818/2744620362 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fa7a00685a0 con 0x7fa784074bc0 2024-04-04T21:39:14.624 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7967fc640 10 client.17642.objecter ms_dispatch 0x7fa7a0000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:39:14.624 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7967fc640 10 client.17642.objecter _finish_command 2 = system:0 2024-04-04T21:39:14.624 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7967fc640 20 client.17642.objecter put_session s=0x7fa7840744f0 osd=0 4 2024-04-04T21:39:14.624 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7967fc640 15 client.17642.objecter _session_command_op_remove 0 2 2024-04-04T21:39:14.625 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-04T21:39:14.625 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-04T21:39:14.625 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-04T21:39:14.628 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 10 librados: watch_flush enter 2024-04-04T21:39:14.629 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 10 librados: watch_flush exit 2024-04-04T21:39:14.629 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 10 client.17642.objecter close_session for osd.0 2024-04-04T21:39:14.629 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 1 -- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:6818/2744620362,v1:172.21.15.106:6819/2744620362] conn(0x7fa784074bc0 msgr2=0x7fa784077060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:14.629 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 1 --2- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:6818/2744620362,v1:172.21.15.106:6819/2744620362] conn(0x7fa784074bc0 0x7fa784077060 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:14.629 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 20 client.17642.objecter put_session s=0x7fa7840744f0 osd=0 1 2024-04-04T21:39:14.629 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 20 client.17642.objecter shutdown clearing up homeless session... 2024-04-04T21:39:14.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 10 client.17642.objecter successfully canceled tick 2024-04-04T21:39:14.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 1 -- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa78406ceb0 msgr2=0x7fa78406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:14.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 1 --2- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fa78406ceb0 0x7fa78406f370 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa7a013bad0 tx=0x7fa790007e80 comp rx=0 tx=0).stop 2024-04-04T21:39:14.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 10 monclient: shutdown 2024-04-04T21:39:14.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:14.630 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 1 -- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7a00b1c00 msgr2=0x7fa7a006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:14.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 1 --2- 172.21.15.106:0/2639185336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7a00b1c00 0x7fa7a006d6b0 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7fa79c00c410 tx=0x7fa79c00c8d0 comp rx=0 tx=0).stop 2024-04-04T21:39:14.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a65e4640 1 -- 172.21.15.106:0/2639185336 reap_dead start 2024-04-04T21:39:14.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 1 -- 172.21.15.106:0/2639185336 shutdown_connections 2024-04-04T21:39:14.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 1 -- 172.21.15.106:0/2639185336 >> 172.21.15.106:0/2639185336 conn(0x7fa7a00b2a80 msgr2=0x7fa7a013dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:14.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 1 -- 172.21.15.106:0/2639185336 shutdown_connections 2024-04-04T21:39:14.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.624+0000 7fa7a8047640 1 -- 172.21.15.106:0/2639185336 wait complete. 2024-04-04T21:39:14.631 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.625+0000 7fa7a8047640 1 librados: shutdown 2024-04-04T21:39:14.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-04T21:39:14.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.739+0000 7fd734128640 10 monclient: get_monmap_and_config 2024-04-04T21:39:14.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.739+0000 7fd734128640 10 monclient: build_initial_monmap 2024-04-04T21:39:14.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.739+0000 7fd734128640 10 monclient: monmap: 2024-04-04T21:39:14.739 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:14.739 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:14.739 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:14.739569+0000 2024-04-04T21:39:14.739 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:14.739569+0000 2024-04-04T21:39:14.739 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:14.739 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:14.739 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:14.739 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:14.739 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:14.739 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:14.739 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:14.739 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:14.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 1 Processor -- start 2024-04-04T21:39:14.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 1 -- start start 2024-04-04T21:39:14.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 10 monclient: init 2024-04-04T21:39:14.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:14.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-04-04T21:39:14.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd72c070cf0 0x7fd72c143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:14.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 10 monclient(hunting): picked mon.noname-c con 0x7fd72c070cf0 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:14.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd72c0b07c0 0x7fd72c13d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:14.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 10 monclient(hunting): picked mon.noname-b con 0x7fd72c0b07c0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:14.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd72c0710d0 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:14.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd72c0a9530 con 0x7fd72c0710d0 2024-04-04T21:39:14.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:14.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd72c072f40 con 0x7fd72c070cf0 2024-04-04T21:39:14.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:14.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c143ad0 con 0x7fd72c0b07c0 2024-04-04T21:39:14.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.740+0000 7fd734128640 10 monclient(hunting): authenticate will time out at 3604.007109s 2024-04-04T21:39:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.741+0000 7fd731ec4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd72c070cf0 0x7fd72c143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.741+0000 7fd7316c3640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd72c0b07c0 0x7fd72c13d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.741+0000 7fd7326c5640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fd72c0710d0 0x7fd72c140410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:43828/0 (socket says 172.21.15.106:43828) 2024-04-04T21:39:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.741+0000 7fd7326c5640 1 -- 172.21.15.106:0/3310253355 learned_addr learned my addr 172.21.15.106:0/3310253355 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:39:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.741+0000 7fd731ec4640 10 monclient(hunting): get_auth_request con 0x7fd72c070cf0 auth_method 0 2024-04-04T21:39:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.741+0000 7fd731ec4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.741+0000 7fd731ec4640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.741+0000 7fd731ec4640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.741+0000 7fd7316c3640 10 monclient(hunting): get_auth_request con 0x7fd72c0b07c0 auth_method 0 2024-04-04T21:39:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.741+0000 7fd7316c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.741+0000 7fd7316c3640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.741+0000 7fd7316c3640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.742+0000 7fd731ec4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:14.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.742+0000 7fd731ec4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:14.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.742+0000 7fd731ec4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:14.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.742+0000 7fd7316c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:14.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.742+0000 7fd7316c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:14.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.742+0000 7fd7316c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:14.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.743+0000 7fd731ec4640 10 monclient(hunting): handle_auth_done global_id 22054 payload 274 2024-04-04T21:39:14.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.743+0000 7fd731ec4640 1 -- 172.21.15.106:0/3310253355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd72c0b07c0 msgr2=0x7fd72c13d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:14.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.743+0000 7fd731ec4640 1 --2- 172.21.15.106:0/3310253355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd72c0b07c0 0x7fd72c13d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:14.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.743+0000 7fd731ec4640 1 -- 172.21.15.106:0/3310253355 >> v1:172.21.15.106:6789/0 conn(0x7fd72c0710d0 legacy=0x7fd72c140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:14.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.743+0000 7fd731ec4640 10 monclient: _finish_hunting 0 2024-04-04T21:39:14.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.743+0000 7fd731ec4640 1 monclient: found mon.noname-c 2024-04-04T21:39:14.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.743+0000 7fd731ec4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:14.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.743+0000 7fd731ec4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:39:14.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.743+0000 7fd731ec4640 1 -- 172.21.15.106:0/3310253355 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72c071f50 con 0x7fd72c070cf0 2024-04-04T21:39:14.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.745+0000 7fd731ec4640 1 --2- 172.21.15.106:0/3310253355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c070cf0 0x7fd72c143ff0 secure :-1 s=READY pgs=5534 cs=0 l=1 rev1=1 crypto rx=0x7fd7280097a0 tx=0x7fd72802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f082033a0d44c84 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:14.750 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd730ec2640 1 -- 172.21.15.106:0/3310253355 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd728035020 con 0x7fd72c070cf0 2024-04-04T21:39:14.750 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd730ec2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:14.750 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd730ec2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:39:14.750 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd730ec2640 10 monclient: dump: 2024-04-04T21:39:14.750 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:14.750 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:14.750 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.750 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.750 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:14.750 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:14.751 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:14.751 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:14.751 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:14.751 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:14.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd730ec2640 10 monclient: _finish_auth 0 2024-04-04T21:39:14.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd730ec2640 10 monclient: _check_auth_tickets 2024-04-04T21:39:14.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd730ec2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:14.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd730ec2640 1 -- 172.21.15.106:0/3310253355 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd728031830 con 0x7fd72c070cf0 2024-04-04T21:39:14.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd734128640 5 monclient: authenticate success, global_id 22054 2024-04-04T21:39:14.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd734128640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:39:14.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd730ec2640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:14.751 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd734128640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:14.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd734128640 4 set_mon_vals no callback set 2024-04-04T21:39:14.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd730ec2640 1 -- 172.21.15.106:0/3310253355 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd728031c00 con 0x7fd72c070cf0 2024-04-04T21:39:14.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd730ec2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:14.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd730ec2640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:14.752 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd730ec2640 10 monclient: dump: 2024-04-04T21:39:14.752 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:14.752 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:14.752 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.752 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.752 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:14.752 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:14.752 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:14.752 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:14.752 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:14.753 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:14.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd734128640 10 monclient: shutdown 2024-04-04T21:39:14.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd734128640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:14.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.749+0000 7fd734128640 1 -- 172.21.15.106:0/3310253355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c070cf0 msgr2=0x7fd72c143ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:14.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.750+0000 7fd734128640 1 --2- 172.21.15.106:0/3310253355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c070cf0 0x7fd72c143ff0 secure :-1 s=READY pgs=5534 cs=0 l=1 rev1=1 crypto rx=0x7fd7280097a0 tx=0x7fd72802b8d0 comp rx=0 tx=0).stop 2024-04-04T21:39:14.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.750+0000 7fd734128640 1 -- 172.21.15.106:0/3310253355 shutdown_connections 2024-04-04T21:39:14.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.750+0000 7fd734128640 1 --2- 172.21.15.106:0/3310253355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd72c0b07c0 0x7fd72c13d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:14.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.750+0000 7fd734128640 1 --2- 172.21.15.106:0/3310253355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c070cf0 0x7fd72c143ff0 unknown :-1 s=CLOSED pgs=5534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:14.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.750+0000 7fd734128640 1 -- 172.21.15.106:0/3310253355 >> 172.21.15.106:0/3310253355 conn(0x7fd72c0b5fa0 msgr2=0x7fd72c0b63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:14.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.750+0000 7fd734128640 1 -- 172.21.15.106:0/3310253355 shutdown_connections 2024-04-04T21:39:14.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.750+0000 7fd734128640 1 -- 172.21.15.106:0/3310253355 wait complete. 2024-04-04T21:39:14.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.750+0000 7fd734128640 10 monclient: build_initial_monmap 2024-04-04T21:39:14.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.750+0000 7fd734128640 10 monclient: monmap: 2024-04-04T21:39:14.753 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:14.754 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:14.754 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:14.754 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:14.754 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:14.754 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:14.754 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:14.754 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:14.754 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:14.754 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:14.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 1 librados: starting msgr at 2024-04-04T21:39:14.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 1 librados: starting objecter 2024-04-04T21:39:14.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 1 Processor -- start 2024-04-04T21:39:14.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 1 -- start start 2024-04-04T21:39:14.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 1 librados: setting wanted keys 2024-04-04T21:39:14.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 1 librados: calling monclient init 2024-04-04T21:39:14.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 10 monclient: init 2024-04-04T21:39:14.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:14.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-04T21:39:14.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c070cf0 0x7fd72c0aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:14.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 10 monclient(hunting): picked mon.noname-b con 0x7fd72c070cf0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:14.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd72c0710d0 0x7fd72c1ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:14.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 10 monclient(hunting): picked mon.noname-c con 0x7fd72c0710d0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:14.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd72c0b07c0 0x7fd72c136e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:14.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 10 monclient(hunting): picked mon.noname-a con 0x7fd72c0b07c0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:14.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:14.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c13db70 con 0x7fd72c0b07c0 2024-04-04T21:39:14.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:14.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c1447c0 con 0x7fd72c070cf0 2024-04-04T21:39:14.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:14.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c13dcf0 con 0x7fd72c0710d0 2024-04-04T21:39:14.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:14.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.751+0000 7fd734128640 10 monclient(hunting): authenticate will time out at 3604.017867s 2024-04-04T21:39:14.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd7326c5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd72c0b07c0 0x7fd72c136e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:14.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd731ec4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c070cf0 0x7fd72c0aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:14.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd7316c3640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd72c0710d0 0x7fd72c1ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:14.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd731ec4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c070cf0 0x7fd72c0aeca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50214/0 (socket says 172.21.15.106:50214) 2024-04-04T21:39:14.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd731ec4640 1 -- 172.21.15.106:0/1781430934 learned_addr learned my addr 172.21.15.106:0/1781430934 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:14.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd731ec4640 10 monclient(hunting): get_auth_request con 0x7fd72c070cf0 auth_method 0 2024-04-04T21:39:14.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd731ec4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:14.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd731ec4640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:14.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd731ec4640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:14.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd7326c5640 10 monclient(hunting): get_auth_request con 0x7fd72c0b07c0 auth_method 0 2024-04-04T21:39:14.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd7326c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:14.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd7326c5640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:14.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd7326c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:14.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd7316c3640 10 monclient(hunting): get_auth_request con 0x7fd72c0710d0 auth_method 0 2024-04-04T21:39:14.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd7316c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:14.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd7316c3640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:14.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.752+0000 7fd7316c3640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:14.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.753+0000 7fd731ec4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:14.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.753+0000 7fd731ec4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:14.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.753+0000 7fd731ec4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:14.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.753+0000 7fd7326c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:14.760 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.753+0000 7fd7326c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:14.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.753+0000 7fd7326c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:14.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.753+0000 7fd7316c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:14.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.753+0000 7fd7316c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:14.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.753+0000 7fd7316c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:14.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.755+0000 7fd731ec4640 10 monclient(hunting): handle_auth_done global_id 22057 payload 824 2024-04-04T21:39:14.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.755+0000 7fd731ec4640 1 -- 172.21.15.106:0/1781430934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd72c0710d0 msgr2=0x7fd72c1ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:14.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.755+0000 7fd731ec4640 1 --2- 172.21.15.106:0/1781430934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd72c0710d0 0x7fd72c1ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:14.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.755+0000 7fd731ec4640 1 -- 172.21.15.106:0/1781430934 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd72c0b07c0 msgr2=0x7fd72c136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:14.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.755+0000 7fd731ec4640 1 --2- 172.21.15.106:0/1781430934 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd72c0b07c0 0x7fd72c136e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:14.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.755+0000 7fd731ec4640 10 monclient: _finish_hunting 0 2024-04-04T21:39:14.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.755+0000 7fd731ec4640 1 monclient: found mon.noname-b 2024-04-04T21:39:14.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.755+0000 7fd731ec4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:14.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.755+0000 7fd731ec4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:14.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.755+0000 7fd731ec4640 1 -- 172.21.15.106:0/1781430934 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72c1c9e60 con 0x7fd72c070cf0 2024-04-04T21:39:14.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.757+0000 7fd731ec4640 1 --2- 172.21.15.106:0/1781430934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c070cf0 0x7fd72c0aeca0 secure :-1 s=READY pgs=5535 cs=0 l=1 rev1=1 crypto rx=0x7fd72802bee0 tx=0x7fd7280046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:14.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.757+0000 7fd7227fc640 10 client.?.objecter ms_handle_connect 0x7fd72c070cf0 2024-04-04T21:39:14.762 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.757+0000 7fd7227fc640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:14.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd7227fc640 1 -- 172.21.15.106:0/1781430934 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd728035040 con 0x7fd72c070cf0 2024-04-04T21:39:14.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd7227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:14.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd7227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:14.763 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd7227fc640 10 monclient: dump: 2024-04-04T21:39:14.763 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:14.763 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:14.763 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.763 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.763 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:14.764 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:14.764 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:14.764 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:14.764 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:14.764 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:14.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd7227fc640 10 monclient: _finish_auth 0 2024-04-04T21:39:14.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd7227fc640 10 monclient: _check_auth_tickets 2024-04-04T21:39:14.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd7227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:14.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd7227fc640 1 -- 172.21.15.106:0/1781430934 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72803b920 con 0x7fd72c070cf0 2024-04-04T21:39:14.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd7227fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:14.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd7227fc640 1 -- 172.21.15.106:0/1781430934 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72803bd10 con 0x7fd72c070cf0 2024-04-04T21:39:14.764 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd7227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:14.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd7227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:14.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd7227fc640 10 monclient: dump: 2024-04-04T21:39:14.765 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:14.765 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:14.765 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.765 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:14.766 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:14.766 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:14.766 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:14.766 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:14.766 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:14.766 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:14.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd734128640 5 monclient: authenticate success, global_id 22057 2024-04-04T21:39:14.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd734128640 10 monclient: _renew_subs 2024-04-04T21:39:14.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd734128640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:14.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd723fff640 4 set_mon_vals no callback set 2024-04-04T21:39:14.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd734128640 1 -- 172.21.15.106:0/1781430934 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd72c1cd0f0 con 0x7fd72c070cf0 2024-04-04T21:39:14.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd734128640 10 client.22057.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:14.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd734128640 10 monclient: _renew_subs 2024-04-04T21:39:14.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd734128640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:14.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd734128640 1 -- 172.21.15.106:0/1781430934 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd72c1cd410 con 0x7fd72c070cf0 2024-04-04T21:39:14.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.761+0000 7fd734128640 1 librados: init done 2024-04-04T21:39:14.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.762+0000 7fd734128640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:14.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.762+0000 7fd734128640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:14.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.762+0000 7fd734128640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:14.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.762+0000 7fd734128640 1 -- 172.21.15.106:0/1781430934 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd72c1cc510 con 0x7fd72c070cf0 2024-04-04T21:39:14.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.779+0000 7fd7227fc640 1 -- 172.21.15.106:0/1781430934 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd728038020 con 0x7fd72c070cf0 2024-04-04T21:39:14.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.780+0000 7fd7227fc640 1 --2- 172.21.15.106:0/1781430934 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd71006ce60 0x7fd71006f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:14.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.780+0000 7fd7316c3640 1 --2- 172.21.15.106:0/1781430934 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd71006ce60 0x7fd71006f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:14.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.780+0000 7fd7316c3640 10 monclient: get_auth_request con 0x7fd71006ce60 auth_method 0 2024-04-04T21:39:14.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.780+0000 7fd7227fc640 10 client.22057.objecter ms_handle_connect 0x7fd71006ce60 2024-04-04T21:39:14.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.780+0000 7fd7316c3640 1 --2- 172.21.15.106:0/1781430934 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd71006ce60 0x7fd71006f320 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd72c1cb870 tx=0x7fd71c009190 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:14.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.782+0000 7fd7227fc640 1 -- 172.21.15.106:0/1781430934 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7fd728078cc0 con 0x7fd72c070cf0 2024-04-04T21:39:14.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.782+0000 7fd7227fc640 10 client.22057.objecter ms_dispatch 0x7fd72c000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:14.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.782+0000 7fd7227fc640 3 client.22057.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:14.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.782+0000 7fd7227fc640 3 client.22057.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:14.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:14.782+0000 7fd7227fc640 20 client.22057.objecter dump_active .. 0 homeless 2024-04-04T21:39:15.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.024+0000 7fd7227fc640 1 -- 172.21.15.106:0/1781430934 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd728031800 con 0x7fd72c070cf0 2024-04-04T21:39:15.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.024+0000 7fd7227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:15.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.024+0000 7fd7227fc640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:15.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.217+0000 7fd734128640 1 -- 172.21.15.106:0/1781430934 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fd72c04f050 con 0x7fd71006ce60 2024-04-04T21:39:15.218 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.217+0000 7fd7227fc640 1 -- 172.21.15.106:0/1781430934 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd72c04f050 con 0x7fd71006ce60 2024-04-04T21:39:15.218 INFO:teuthology.orchestra.run.smithi106.stderr:instructed osd(s) 0 to deep-scrub 2024-04-04T21:39:15.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 10 librados: watch_flush enter 2024-04-04T21:39:15.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 10 librados: watch_flush exit 2024-04-04T21:39:15.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 20 client.22057.objecter shutdown clearing up homeless session... 2024-04-04T21:39:15.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 10 client.22057.objecter successfully canceled tick 2024-04-04T21:39:15.226 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 1 -- 172.21.15.106:0/1781430934 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd71006ce60 msgr2=0x7fd71006f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:15.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 1 --2- 172.21.15.106:0/1781430934 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd71006ce60 0x7fd71006f320 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd72c1cb870 tx=0x7fd71c009190 comp rx=0 tx=0).stop 2024-04-04T21:39:15.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 10 monclient: shutdown 2024-04-04T21:39:15.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:15.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 1 -- 172.21.15.106:0/1781430934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c070cf0 msgr2=0x7fd72c0aeca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:15.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 1 --2- 172.21.15.106:0/1781430934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c070cf0 0x7fd72c0aeca0 secure :-1 s=READY pgs=5535 cs=0 l=1 rev1=1 crypto rx=0x7fd72802bee0 tx=0x7fd7280046c0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 1 -- 172.21.15.106:0/1781430934 shutdown_connections 2024-04-04T21:39:15.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 1 --2- 172.21.15.106:0/1781430934 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fd71006ce60 0x7fd71006f320 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 1 --2- 172.21.15.106:0/1781430934 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd72c0b07c0 0x7fd72c136e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 1 --2- 172.21.15.106:0/1781430934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd72c0710d0 0x7fd72c1ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.227 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 1 --2- 172.21.15.106:0/1781430934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd72c070cf0 0x7fd72c0aeca0 unknown :-1 s=CLOSED pgs=5535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 1 -- 172.21.15.106:0/1781430934 >> 172.21.15.106:0/1781430934 conn(0x7fd72c0b5fa0 msgr2=0x7fd72c13c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:15.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 1 -- 172.21.15.106:0/1781430934 shutdown_connections 2024-04-04T21:39:15.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 1 -- 172.21.15.106:0/1781430934 wait complete. 2024-04-04T21:39:15.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.219+0000 7fd734128640 1 librados: shutdown 2024-04-04T21:39:15.230 INFO:tasks.ceph:Scrubbing osd.1 2024-04-04T21:39:15.230 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-04T21:39:15.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.343+0000 7fe49cb23640 10 monclient: get_monmap_and_config 2024-04-04T21:39:15.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.343+0000 7fe49cb23640 10 monclient: build_initial_monmap 2024-04-04T21:39:15.343 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.343+0000 7fe49cb23640 10 monclient: monmap: 2024-04-04T21:39:15.344 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:15.344 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:15.344 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:15.343416+0000 2024-04-04T21:39:15.344 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:15.343416+0000 2024-04-04T21:39:15.344 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:15.344 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:15.344 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:15.344 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:15.344 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:15.344 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:15.344 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:15.345 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:15.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.343+0000 7fe49cb23640 1 Processor -- start 2024-04-04T21:39:15.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.343+0000 7fe49cb23640 1 -- start start 2024-04-04T21:39:15.345 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.343+0000 7fe49cb23640 10 monclient: init 2024-04-04T21:39:15.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.344+0000 7fe49cb23640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:15.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.344+0000 7fe49cb23640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-04-04T21:39:15.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.344+0000 7fe49cb23640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4980b5470 addr v1:172.21.15.203:6789/0 2024-04-04T21:39:15.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.344+0000 7fe49cb23640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe4980aa3a0 0x7fe49813f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:15.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.344+0000 7fe49cb23640 10 monclient(hunting): picked mon.noname-c con 0x7fe4980aa3a0 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:15.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.344+0000 7fe49cb23640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe49813fdd0 0x7fe49814cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:15.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.344+0000 7fe49cb23640 10 monclient(hunting): picked mon.noname-b con 0x7fe49813fdd0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:15.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.344+0000 7fe49cb23640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4981401f0 con 0x7fe4980b5470 2024-04-04T21:39:15.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.344+0000 7fe49cb23640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:15.347 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.344+0000 7fe49cb23640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4980b5850 con 0x7fe4980aa3a0 2024-04-04T21:39:15.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.344+0000 7fe49cb23640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:15.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.344+0000 7fe49cb23640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4980a9f10 con 0x7fe49813fdd0 2024-04-04T21:39:15.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.344+0000 7fe49cb23640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:15.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.344+0000 7fe49cb23640 10 monclient(hunting): authenticate will time out at 3604.610903s 2024-04-04T21:39:15.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.345+0000 7fe496d9d640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe49813fdd0 0x7fe49814cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:15.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.345+0000 7fe495d9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe4980aa3a0 0x7fe49813f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:15.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.345+0000 7fe496d9d640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe49813fdd0 0x7fe49814cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:38092/0 (socket says 172.21.15.106:38092) 2024-04-04T21:39:15.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.345+0000 7fe496d9d640 1 -- 172.21.15.106:0/224042485 learned_addr learned my addr 172.21.15.106:0/224042485 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:15.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.345+0000 7fe496d9d640 10 monclient(hunting): get_auth_request con 0x7fe49813fdd0 auth_method 0 2024-04-04T21:39:15.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.345+0000 7fe496d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:15.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.345+0000 7fe496d9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:15.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.345+0000 7fe496d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:15.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.346+0000 7fe495d9b640 10 monclient(hunting): get_auth_request con 0x7fe4980aa3a0 auth_method 0 2024-04-04T21:39:15.348 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.346+0000 7fe495d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.346+0000 7fe495d9b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.346+0000 7fe495d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.346+0000 7fe496d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.346+0000 7fe496d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.346+0000 7fe496d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.346+0000 7fe495d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.346+0000 7fe495d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.346+0000 7fe495d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.347+0000 7fe496d9d640 10 monclient(hunting): handle_auth_done global_id 17651 payload 274 2024-04-04T21:39:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.347+0000 7fe496d9d640 1 -- 172.21.15.106:0/224042485 >> v2:172.21.15.203:3300/0 conn(0x7fe4980aa3a0 msgr2=0x7fe49813f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.347+0000 7fe496d9d640 1 --2- 172.21.15.106:0/224042485 >> v2:172.21.15.203:3300/0 conn(0x7fe4980aa3a0 0x7fe49813f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.347+0000 7fe496d9d640 1 -- 172.21.15.106:0/224042485 >> v1:172.21.15.203:6789/0 conn(0x7fe4980b5470 legacy=0x7fe4981425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.347+0000 7fe496d9d640 10 monclient: _finish_hunting 0 2024-04-04T21:39:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.347+0000 7fe496d9d640 1 monclient: found mon.noname-b 2024-04-04T21:39:15.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.347+0000 7fe496d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:15.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.347+0000 7fe496d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:39:15.350 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.347+0000 7fe496d9d640 1 -- 172.21.15.106:0/224042485 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4980ab700 con 0x7fe49813fdd0 2024-04-04T21:39:15.351 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.349+0000 7fe496d9d640 1 --2- 172.21.15.106:0/224042485 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe49813fdd0 0x7fe49814cec0 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7fe490009e50 tx=0x7fe49002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a75111b7d982be5b server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:15.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49559a640 1 -- 172.21.15.106:0/224042485 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49002c9e0 con 0x7fe49813fdd0 2024-04-04T21:39:15.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:15.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:15.354 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49559a640 10 monclient: dump: 2024-04-04T21:39:15.355 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:15.355 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:15.355 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.355 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.355 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:15.355 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:15.356 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:15.356 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:15.356 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:15.356 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:15.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49559a640 10 monclient: _finish_auth 0 2024-04-04T21:39:15.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49559a640 10 monclient: _check_auth_tickets 2024-04-04T21:39:15.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:15.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49559a640 1 -- 172.21.15.106:0/224042485 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49002cb80 con 0x7fe49813fdd0 2024-04-04T21:39:15.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49cb23640 5 monclient: authenticate success, global_id 17651 2024-04-04T21:39:15.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49cb23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:39:15.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49559a640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:15.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49cb23640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:15.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49559a640 1 -- 172.21.15.106:0/224042485 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4900343f0 con 0x7fe49813fdd0 2024-04-04T21:39:15.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49cb23640 4 set_mon_vals no callback set 2024-04-04T21:39:15.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:15.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:39:15.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49559a640 10 monclient: dump: 2024-04-04T21:39:15.358 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:15.359 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:15.359 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.359 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.359 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:15.359 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:15.359 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:15.359 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:15.359 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:15.359 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:15.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49cb23640 10 monclient: shutdown 2024-04-04T21:39:15.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49cb23640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:15.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49cb23640 1 -- 172.21.15.106:0/224042485 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe49813fdd0 msgr2=0x7fe49814cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:15.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.353+0000 7fe49cb23640 1 --2- 172.21.15.106:0/224042485 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe49813fdd0 0x7fe49814cec0 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7fe490009e50 tx=0x7fe49002be40 comp rx=0 tx=0).stop 2024-04-04T21:39:15.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.354+0000 7fe49cb23640 1 -- 172.21.15.106:0/224042485 shutdown_connections 2024-04-04T21:39:15.360 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.354+0000 7fe49cb23640 1 --2- 172.21.15.106:0/224042485 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe49813fdd0 0x7fe49814cec0 unknown :-1 s=CLOSED pgs=3267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.354+0000 7fe49cb23640 1 --2- 172.21.15.106:0/224042485 >> v2:172.21.15.203:3300/0 conn(0x7fe4980aa3a0 0x7fe49813f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.354+0000 7fe49cb23640 1 -- 172.21.15.106:0/224042485 >> 172.21.15.106:0/224042485 conn(0x7fe498066e70 msgr2=0x7fe498067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:15.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.354+0000 7fe49cb23640 1 -- 172.21.15.106:0/224042485 shutdown_connections 2024-04-04T21:39:15.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.354+0000 7fe49cb23640 1 -- 172.21.15.106:0/224042485 wait complete. 2024-04-04T21:39:15.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.354+0000 7fe49cb23640 10 monclient: build_initial_monmap 2024-04-04T21:39:15.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.354+0000 7fe49cb23640 10 monclient: monmap: 2024-04-04T21:39:15.361 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:15.362 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:15.362 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:15.362 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:15.362 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:15.362 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:15.362 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:15.362 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:15.363 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:15.363 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:15.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 1 librados: starting msgr at 2024-04-04T21:39:15.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 1 librados: starting objecter 2024-04-04T21:39:15.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 1 Processor -- start 2024-04-04T21:39:15.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 1 -- start start 2024-04-04T21:39:15.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 1 librados: setting wanted keys 2024-04-04T21:39:15.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 1 librados: calling monclient init 2024-04-04T21:39:15.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 10 monclient: init 2024-04-04T21:39:15.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:15.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-04T21:39:15.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4980aa3a0 0x7fe498146020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:15.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 10 monclient(hunting): picked mon.noname-c con 0x7fe4980aa3a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:15.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4980b5470 0x7fe4981dadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:15.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 10 monclient(hunting): picked mon.noname-a con 0x7fe4980b5470 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:15.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe49813fdd0 0x7fe4980b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:15.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 10 monclient(hunting): picked mon.noname-b con 0x7fe49813fdd0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:15.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:15.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4980a9aa0 con 0x7fe4980b5470 2024-04-04T21:39:15.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:15.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4980b27e0 con 0x7fe49813fdd0 2024-04-04T21:39:15.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:15.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4980a9c20 con 0x7fe4980aa3a0 2024-04-04T21:39:15.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:15.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.355+0000 7fe49cb23640 10 monclient(hunting): authenticate will time out at 3604.622151s 2024-04-04T21:39:15.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.356+0000 7fe495d9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4980b5470 0x7fe4981dadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:15.374 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.356+0000 7fe49659c640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4980aa3a0 0x7fe498146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:15.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.356+0000 7fe496d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe49813fdd0 0x7fe4980b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:15.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.356+0000 7fe496d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe49813fdd0 0x7fe4980b7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50224/0 (socket says 172.21.15.106:50224) 2024-04-04T21:39:15.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.356+0000 7fe496d9d640 1 -- 172.21.15.106:0/1138529191 learned_addr learned my addr 172.21.15.106:0/1138529191 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:15.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.356+0000 7fe496d9d640 10 monclient(hunting): get_auth_request con 0x7fe49813fdd0 auth_method 0 2024-04-04T21:39:15.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.356+0000 7fe496d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:15.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.356+0000 7fe496d9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:15.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.356+0000 7fe496d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:15.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.356+0000 7fe49659c640 10 monclient(hunting): get_auth_request con 0x7fe4980aa3a0 auth_method 0 2024-04-04T21:39:15.375 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.356+0000 7fe49659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:15.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.356+0000 7fe49659c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:15.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.356+0000 7fe49659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:15.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.356+0000 7fe495d9b640 10 monclient(hunting): get_auth_request con 0x7fe4980b5470 auth_method 0 2024-04-04T21:39:15.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.357+0000 7fe495d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:15.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.357+0000 7fe495d9b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:15.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.357+0000 7fe495d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:15.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.357+0000 7fe496d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:15.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.357+0000 7fe496d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:15.376 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.357+0000 7fe496d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:15.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.357+0000 7fe49659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:15.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.357+0000 7fe49659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:15.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.357+0000 7fe49659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:15.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.357+0000 7fe495d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:15.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.357+0000 7fe495d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:15.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.357+0000 7fe495d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:15.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.360+0000 7fe496d9d640 10 monclient(hunting): handle_auth_done global_id 22066 payload 824 2024-04-04T21:39:15.377 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.360+0000 7fe496d9d640 1 -- 172.21.15.106:0/1138529191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4980aa3a0 msgr2=0x7fe498146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:15.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.360+0000 7fe496d9d640 1 --2- 172.21.15.106:0/1138529191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4980aa3a0 0x7fe498146020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.360+0000 7fe496d9d640 1 -- 172.21.15.106:0/1138529191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4980b5470 msgr2=0x7fe4981dadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:15.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.360+0000 7fe496d9d640 1 --2- 172.21.15.106:0/1138529191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4980b5470 0x7fe4981dadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.360+0000 7fe496d9d640 10 monclient: _finish_hunting 0 2024-04-04T21:39:15.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.360+0000 7fe496d9d640 1 monclient: found mon.noname-b 2024-04-04T21:39:15.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.360+0000 7fe496d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:15.378 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.360+0000 7fe496d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:15.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.360+0000 7fe496d9d640 1 -- 172.21.15.106:0/1138529191 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4981cc460 con 0x7fe49813fdd0 2024-04-04T21:39:15.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.362+0000 7fe496d9d640 1 --2- 172.21.15.106:0/1138529191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe49813fdd0 0x7fe4980b7a00 secure :-1 s=READY pgs=5537 cs=0 l=1 rev1=1 crypto rx=0x7fe490036040 tx=0x7fe490004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:15.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.362+0000 7fe486ffd640 10 client.?.objecter ms_handle_connect 0x7fe49813fdd0 2024-04-04T21:39:15.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.362+0000 7fe486ffd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:15.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe486ffd640 1 -- 172.21.15.106:0/1138529191 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe490043020 con 0x7fe49813fdd0 2024-04-04T21:39:15.379 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe486ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:15.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe486ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:15.380 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe486ffd640 10 monclient: dump: 2024-04-04T21:39:15.380 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:15.380 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:15.380 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.380 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.381 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:15.381 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:15.381 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:15.381 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:15.381 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:15.381 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:15.381 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe486ffd640 10 monclient: _finish_auth 0 2024-04-04T21:39:15.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe486ffd640 10 monclient: _check_auth_tickets 2024-04-04T21:39:15.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe486ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:15.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe49cb23640 5 monclient: authenticate success, global_id 22066 2024-04-04T21:39:15.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe49cb23640 10 monclient: _renew_subs 2024-04-04T21:39:15.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe49cb23640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:15.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe49cb23640 1 -- 172.21.15.106:0/1138529191 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4981cc740 con 0x7fe49813fdd0 2024-04-04T21:39:15.382 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe49cb23640 10 client.22066.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:15.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe49cb23640 10 monclient: _renew_subs 2024-04-04T21:39:15.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe49cb23640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:15.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe49cb23640 1 -- 172.21.15.106:0/1138529191 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4981dbc20 con 0x7fe49813fdd0 2024-04-04T21:39:15.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.364+0000 7fe49cb23640 1 librados: init done 2024-04-04T21:39:15.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.365+0000 7fe49cb23640 10 client.22066.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:15.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.365+0000 7fe49cb23640 15 client.22066.objecter _session_command_op_assign -1 1 2024-04-04T21:39:15.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.365+0000 7fe49cb23640 20 client.22066.objecter _get_session osd=-1 returning homeless 2024-04-04T21:39:15.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.365+0000 7fe49cb23640 10 client.22066.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:15.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.365+0000 7fe49cb23640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:15.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.365+0000 7fe49cb23640 1 -- 172.21.15.106:0/1138529191 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe4980685a0 con 0x7fe49813fdd0 2024-04-04T21:39:15.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.366+0000 7fe486ffd640 1 -- 172.21.15.106:0/1138529191 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4980685a0 con 0x7fe49813fdd0 2024-04-04T21:39:15.383 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.366+0000 7fe486ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:15.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.366+0000 7fe486ffd640 1 -- 172.21.15.106:0/1138529191 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe490009510 con 0x7fe49813fdd0 2024-04-04T21:39:15.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.366+0000 7fe486ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:15.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.366+0000 7fe486ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:15.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.366+0000 7fe494d99640 4 set_mon_vals no callback set 2024-04-04T21:39:15.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.366+0000 7fe486ffd640 10 monclient: dump: 2024-04-04T21:39:15.384 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:15.384 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:15.384 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.384 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.385 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:15.385 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:15.385 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:15.385 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:15.385 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:15.385 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:15.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.386+0000 7fe486ffd640 1 -- 172.21.15.106:0/1138529191 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe490046020 con 0x7fe49813fdd0 2024-04-04T21:39:15.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.386+0000 7fe486ffd640 1 --2- 172.21.15.106:0/1138529191 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe47806ce60 0x7fe47806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:15.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.387+0000 7fe49659c640 1 --2- 172.21.15.106:0/1138529191 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe47806ce60 0x7fe47806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:15.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.387+0000 7fe49659c640 10 monclient: get_auth_request con 0x7fe47806ce60 auth_method 0 2024-04-04T21:39:15.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.387+0000 7fe49659c640 1 --2- 172.21.15.106:0/1138529191 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe47806ce60 0x7fe47806f320 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe48c000f60 tx=0x7fe48c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:15.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.387+0000 7fe486ffd640 10 client.22066.objecter ms_handle_connect 0x7fe47806ce60 2024-04-04T21:39:15.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 1 -- 172.21.15.106:0/1138529191 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7fe4900792e0 con 0x7fe49813fdd0 2024-04-04T21:39:15.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 10 client.22066.objecter ms_dispatch 0x7fe498000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:15.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 3 client.22066.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:15.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 3 client.22066.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:15.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 10 client.22066.objecter checking command 1 2024-04-04T21:39:15.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 1 --2- 172.21.15.106:0/1138529191 >> [v2:172.21.15.106:6826/2321665741,v1:172.21.15.106:6827/2321665741] conn(0x7fe478074b70 0x7fe478077010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:15.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 20 client.22066.objecter _get_session s=0x7fe4780744a0 osd=1 3 2024-04-04T21:39:15.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 20 client.22066.objecter put_session s=0x7fe4780744a0 osd=1 3 2024-04-04T21:39:15.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 15 client.22066.objecter _session_command_op_remove -1 1 2024-04-04T21:39:15.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 20 client.22066.objecter _get_session s=0x7fe4780744a0 osd=1 3 2024-04-04T21:39:15.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 20 client.22066.objecter get_session s=0x7fe4780744a0 osd=1 3 2024-04-04T21:39:15.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 15 client.22066.objecter _session_command_op_assign 1 1 2024-04-04T21:39:15.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 20 client.22066.objecter put_session s=0x7fe4780744a0 osd=1 4 2024-04-04T21:39:15.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 10 client.22066.objecter _send_command 1 2024-04-04T21:39:15.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 1 -- 172.21.15.106:0/1138529191 --> [v2:172.21.15.106:6826/2321665741,v1:172.21.15.106:6827/2321665741] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe490042e50 con 0x7fe478074b70 2024-04-04T21:39:15.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 20 client.22066.objecter dump_active .. 0 homeless 2024-04-04T21:39:15.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 1 -- 172.21.15.106:0/1138529191 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=1197) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4900774e0 con 0x7fe49813fdd0 2024-04-04T21:39:15.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.388+0000 7fe486ffd640 10 monclient: handle_get_version_reply finishing 1 version 1197 2024-04-04T21:39:15.389 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.389+0000 7fe495d9b640 1 --2- 172.21.15.106:0/1138529191 >> [v2:172.21.15.106:6826/2321665741,v1:172.21.15.106:6827/2321665741] conn(0x7fe478074b70 0x7fe478077010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:15.390 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.389+0000 7fe495d9b640 10 monclient: get_auth_request con 0x7fe478074b70 auth_method 0 2024-04-04T21:39:15.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.392+0000 7fe495d9b640 1 --2- 172.21.15.106:0/1138529191 >> [v2:172.21.15.106:6826/2321665741,v1:172.21.15.106:6827/2321665741] conn(0x7fe478074b70 0x7fe478077010 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:15.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.392+0000 7fe486ffd640 10 client.22066.objecter ms_handle_connect 0x7fe478074b70 2024-04-04T21:39:15.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.416+0000 7fe486ffd640 1 -- 172.21.15.106:0/1138529191 <== osd.1 v2:172.21.15.106:6826/2321665741 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe490042e50 con 0x7fe478074b70 2024-04-04T21:39:15.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.416+0000 7fe486ffd640 10 client.22066.objecter ms_dispatch 0x7fe498000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:39:15.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.416+0000 7fe486ffd640 10 client.22066.objecter _finish_command 1 = system:0 2024-04-04T21:39:15.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.416+0000 7fe486ffd640 20 client.22066.objecter put_session s=0x7fe4780744a0 osd=1 4 2024-04-04T21:39:15.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.416+0000 7fe486ffd640 15 client.22066.objecter _session_command_op_remove 1 1 2024-04-04T21:39:15.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.430+0000 7fe49cb23640 10 client.22066.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-04T21:39:15.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.430+0000 7fe49cb23640 15 client.22066.objecter _session_command_op_assign -1 2 2024-04-04T21:39:15.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.430+0000 7fe49cb23640 20 client.22066.objecter _get_session s=0x7fe4780744a0 osd=1 3 2024-04-04T21:39:15.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.430+0000 7fe49cb23640 20 client.22066.objecter put_session s=0x7fe4780744a0 osd=1 3 2024-04-04T21:39:15.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.430+0000 7fe49cb23640 20 client.22066.objecter _get_session s=0x7fe4780744a0 osd=1 3 2024-04-04T21:39:15.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.430+0000 7fe49cb23640 15 client.22066.objecter _session_command_op_remove -1 2 2024-04-04T21:39:15.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.430+0000 7fe49cb23640 20 client.22066.objecter get_session s=0x7fe4780744a0 osd=1 3 2024-04-04T21:39:15.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.430+0000 7fe49cb23640 15 client.22066.objecter _session_command_op_assign 1 2 2024-04-04T21:39:15.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.430+0000 7fe49cb23640 20 client.22066.objecter put_session s=0x7fe4780744a0 osd=1 4 2024-04-04T21:39:15.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.430+0000 7fe49cb23640 10 client.22066.objecter _send_command 2 2024-04-04T21:39:15.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.430+0000 7fe49cb23640 1 -- 172.21.15.106:0/1138529191 --> [v2:172.21.15.106:6826/2321665741,v1:172.21.15.106:6827/2321665741] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe4981db6c0 con 0x7fe478074b70 2024-04-04T21:39:15.444 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.445+0000 7fe486ffd640 1 -- 172.21.15.106:0/1138529191 <== osd.1 v2:172.21.15.106:6826/2321665741 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe4981db6c0 con 0x7fe478074b70 2024-04-04T21:39:15.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.445+0000 7fe486ffd640 10 client.22066.objecter ms_dispatch 0x7fe498000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:39:15.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.445+0000 7fe486ffd640 10 client.22066.objecter _finish_command 2 = system:0 2024-04-04T21:39:15.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.445+0000 7fe486ffd640 20 client.22066.objecter put_session s=0x7fe4780744a0 osd=1 4 2024-04-04T21:39:15.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.445+0000 7fe486ffd640 15 client.22066.objecter _session_command_op_remove 1 2 2024-04-04T21:39:15.445 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-04T21:39:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-04T21:39:15.445 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-04T21:39:15.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 10 librados: watch_flush enter 2024-04-04T21:39:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 10 librados: watch_flush exit 2024-04-04T21:39:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 10 client.22066.objecter close_session for osd.1 2024-04-04T21:39:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 1 -- 172.21.15.106:0/1138529191 >> [v2:172.21.15.106:6826/2321665741,v1:172.21.15.106:6827/2321665741] conn(0x7fe478074b70 msgr2=0x7fe478077010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 1 --2- 172.21.15.106:0/1138529191 >> [v2:172.21.15.106:6826/2321665741,v1:172.21.15.106:6827/2321665741] conn(0x7fe478074b70 0x7fe478077010 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 20 client.22066.objecter put_session s=0x7fe4780744a0 osd=1 1 2024-04-04T21:39:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 20 client.22066.objecter shutdown clearing up homeless session... 2024-04-04T21:39:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 10 client.22066.objecter successfully canceled tick 2024-04-04T21:39:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 1 -- 172.21.15.106:0/1138529191 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe47806ce60 msgr2=0x7fe47806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 1 --2- 172.21.15.106:0/1138529191 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe47806ce60 0x7fe47806f320 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe48c000f60 tx=0x7fe48c009040 comp rx=0 tx=0).stop 2024-04-04T21:39:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 10 monclient: shutdown 2024-04-04T21:39:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 1 -- 172.21.15.106:0/1138529191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe49813fdd0 msgr2=0x7fe4980b7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 1 --2- 172.21.15.106:0/1138529191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe49813fdd0 0x7fe4980b7a00 secure :-1 s=READY pgs=5537 cs=0 l=1 rev1=1 crypto rx=0x7fe490036040 tx=0x7fe490004a40 comp rx=0 tx=0).stop 2024-04-04T21:39:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe496d9d640 1 -- 172.21.15.106:0/1138529191 reap_dead start 2024-04-04T21:39:15.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 1 -- 172.21.15.106:0/1138529191 shutdown_connections 2024-04-04T21:39:15.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 1 -- 172.21.15.106:0/1138529191 >> 172.21.15.106:0/1138529191 conn(0x7fe498066e70 msgr2=0x7fe498067250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:15.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 1 -- 172.21.15.106:0/1138529191 shutdown_connections 2024-04-04T21:39:15.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 1 -- 172.21.15.106:0/1138529191 wait complete. 2024-04-04T21:39:15.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.446+0000 7fe49cb23640 1 librados: shutdown 2024-04-04T21:39:15.456 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-04T21:39:15.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.556+0000 7f14ddb93640 10 monclient: get_monmap_and_config 2024-04-04T21:39:15.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.556+0000 7f14ddb93640 10 monclient: build_initial_monmap 2024-04-04T21:39:15.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.556+0000 7f14ddb93640 10 monclient: monmap: 2024-04-04T21:39:15.557 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:15.557 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:15.557 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:15.557912+0000 2024-04-04T21:39:15.557 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:15.557912+0000 2024-04-04T21:39:15.557 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:15.557 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:15.557 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:15.558 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:15.558 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:15.558 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:15.558 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:15.558 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:15.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.557+0000 7f14ddb93640 1 Processor -- start 2024-04-04T21:39:15.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.557+0000 7f14ddb93640 1 -- start start 2024-04-04T21:39:15.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.557+0000 7f14ddb93640 10 monclient: init 2024-04-04T21:39:15.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14ddb93640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:15.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14ddb93640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-04-04T21:39:15.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14ddb93640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f14d80b5200 0x7f14d813f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:15.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14ddb93640 10 monclient(hunting): picked mon.noname-a con 0x7f14d80b5200 addr v2:172.21.15.106:3300/0 2024-04-04T21:39:15.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14ddb93640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f14d80a9aa0 0x7f14d8145af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:15.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14ddb93640 10 monclient(hunting): picked mon.noname-c con 0x7f14d80a9aa0 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:15.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14ddb93640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14d813fef0 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:15.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14ddb93640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d80a97e0 con 0x7f14d813fef0 2024-04-04T21:39:15.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14ddb93640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:15.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14ddb93640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d80b55e0 con 0x7f14d80b5200 2024-04-04T21:39:15.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14ddb93640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:15.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14ddb93640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d80aa070 con 0x7f14d80a9aa0 2024-04-04T21:39:15.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14ddb93640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14ddb93640 10 monclient(hunting): authenticate will time out at 3604.825399s 2024-04-04T21:39:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14d77fe640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f14d80b5200 0x7f14d813f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.558+0000 7f14d6ffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f14d80a9aa0 0x7f14d8145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d6ffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f14d80a9aa0 0x7f14d8145af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50232/0 (socket says 172.21.15.106:50232) 2024-04-04T21:39:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d6ffd640 1 -- 172.21.15.106:0/2015058821 learned_addr learned my addr 172.21.15.106:0/2015058821 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d6ffd640 10 monclient(hunting): get_auth_request con 0x7f14d80a9aa0 auth_method 0 2024-04-04T21:39:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d77fe640 10 monclient(hunting): get_auth_request con 0x7f14d80b5200 auth_method 0 2024-04-04T21:39:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.559+0000 7f14d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:15.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.561+0000 7f14d77fe640 10 monclient(hunting): handle_auth_done global_id 15120 payload 274 2024-04-04T21:39:15.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.561+0000 7f14d77fe640 1 -- 172.21.15.106:0/2015058821 >> v2:172.21.15.203:3300/0 conn(0x7f14d80a9aa0 msgr2=0x7f14d8145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:15.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.561+0000 7f14d77fe640 1 --2- 172.21.15.106:0/2015058821 >> v2:172.21.15.203:3300/0 conn(0x7f14d80a9aa0 0x7f14d8145af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.561+0000 7f14d77fe640 1 -- 172.21.15.106:0/2015058821 >> v1:172.21.15.106:6789/0 conn(0x7f14d813fef0 legacy=0x7f14d80b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:15.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.561+0000 7f14d77fe640 10 monclient: _finish_hunting 0 2024-04-04T21:39:15.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.561+0000 7f14d77fe640 1 monclient: found mon.noname-a 2024-04-04T21:39:15.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.561+0000 7f14d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:15.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.561+0000 7f14d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:39:15.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.561+0000 7f14d77fe640 1 -- 172.21.15.106:0/2015058821 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d80ab420 con 0x7f14d80b5200 2024-04-04T21:39:15.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.561+0000 7f14d6ffd640 1 --2- 172.21.15.106:0/2015058821 >> v2:172.21.15.203:3300/0 conn(0x7f14d80a9aa0 0x7f14d8145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-04T21:39:15.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.564+0000 7f14d77fe640 1 --2- 172.21.15.106:0/2015058821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14d80b5200 0x7f14d813f9b0 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7f14d80679d0 tx=0x7f14c802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd9c44870652ce9 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:15.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14d67fc640 1 -- 172.21.15.106:0/2015058821 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c8038070 con 0x7f14d80b5200 2024-04-04T21:39:15.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:15.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14d67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:39:15.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14d67fc640 10 monclient: dump: 2024-04-04T21:39:15.568 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:15.568 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:15.568 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.568 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.568 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:15.568 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:15.568 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:15.568 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14d67fc640 10 monclient: _finish_auth 0 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14d67fc640 10 monclient: _check_auth_tickets 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14d67fc640 1 -- 172.21.15.106:0/2015058821 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c8031710 con 0x7f14d80b5200 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14d67fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14d67fc640 1 -- 172.21.15.106:0/2015058821 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c8031ae0 con 0x7f14d80b5200 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14d67fc640 10 monclient: dump: 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.569 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.570 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:15.570 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:15.570 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:15.570 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:15.570 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:15.570 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:15.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14ddb93640 5 monclient: authenticate success, global_id 15120 2024-04-04T21:39:15.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14ddb93640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:15.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.567+0000 7f14ddb93640 4 set_mon_vals no callback set 2024-04-04T21:39:15.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.568+0000 7f14ddb93640 10 monclient: shutdown 2024-04-04T21:39:15.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.568+0000 7f14ddb93640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:15.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.568+0000 7f14ddb93640 1 -- 172.21.15.106:0/2015058821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14d80b5200 msgr2=0x7f14d813f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.568+0000 7f14ddb93640 1 --2- 172.21.15.106:0/2015058821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14d80b5200 0x7f14d813f9b0 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7f14d80679d0 tx=0x7f14c802b7b0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.568+0000 7f14ddb93640 1 -- 172.21.15.106:0/2015058821 shutdown_connections 2024-04-04T21:39:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.568+0000 7f14ddb93640 1 --2- 172.21.15.106:0/2015058821 >> v2:172.21.15.203:3300/0 conn(0x7f14d80a9aa0 0x7f14d8145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.568+0000 7f14ddb93640 1 --2- 172.21.15.106:0/2015058821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14d80b5200 0x7f14d813f9b0 unknown :-1 s=CLOSED pgs=3161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.568+0000 7f14ddb93640 1 -- 172.21.15.106:0/2015058821 >> 172.21.15.106:0/2015058821 conn(0x7f14d8066e40 msgr2=0x7f14d8067250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.568+0000 7f14ddb93640 1 -- 172.21.15.106:0/2015058821 shutdown_connections 2024-04-04T21:39:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.568+0000 7f14ddb93640 1 -- 172.21.15.106:0/2015058821 wait complete. 2024-04-04T21:39:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.569+0000 7f14ddb93640 10 monclient: build_initial_monmap 2024-04-04T21:39:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.569+0000 7f14ddb93640 10 monclient: monmap: 2024-04-04T21:39:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.569+0000 7f14ddb93640 1 librados: starting msgr at 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.569+0000 7f14ddb93640 1 librados: starting objecter 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.569+0000 7f14ddb93640 1 Processor -- start 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.569+0000 7f14ddb93640 1 -- start start 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.569+0000 7f14ddb93640 1 librados: setting wanted keys 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.569+0000 7f14ddb93640 1 librados: calling monclient init 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.569+0000 7f14ddb93640 10 monclient: init 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-04T21:39:15.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14d80a9aa0 0x7f14d81cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:15.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 10 monclient(hunting): picked mon.noname-a con 0x7f14d80a9aa0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:15.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14d80b5200 0x7f14d81ce570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:15.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 10 monclient(hunting): picked mon.noname-b con 0x7f14d80b5200 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:15.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14d813fef0 0x7f14d81ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:15.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 10 monclient(hunting): picked mon.noname-c con 0x7f14d813fef0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:15.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:15.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d80aa680 con 0x7f14d80a9aa0 2024-04-04T21:39:15.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:15.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d80aa500 con 0x7f14d80b5200 2024-04-04T21:39:15.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:15.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d8146030 con 0x7f14d813fef0 2024-04-04T21:39:15.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14ddb93640 10 monclient(hunting): authenticate will time out at 3604.837458s 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14d77fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14d80a9aa0 0x7f14d81cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14d6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14d80b5200 0x7f14d81ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.570+0000 7f14d7fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14d813fef0 0x7f14d81ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d77fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14d80a9aa0 0x7f14d81cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42422/0 (socket says 172.21.15.106:42422) 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d77fe640 1 -- 172.21.15.106:0/274009395 learned_addr learned my addr 172.21.15.106:0/274009395 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d77fe640 10 monclient(hunting): get_auth_request con 0x7f14d80a9aa0 auth_method 0 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d6ffd640 10 monclient(hunting): get_auth_request con 0x7f14d80b5200 auth_method 0 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:15.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d7fff640 10 monclient(hunting): get_auth_request con 0x7f14d813fef0 auth_method 0 2024-04-04T21:39:15.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:15.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d7fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:15.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:15.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:15.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:15.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:15.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:15.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:15.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:15.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:15.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:15.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.571+0000 7f14d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:15.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.574+0000 7f14d6ffd640 10 monclient(hunting): handle_auth_done global_id 22072 payload 824 2024-04-04T21:39:15.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.574+0000 7f14d6ffd640 1 -- 172.21.15.106:0/274009395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14d813fef0 msgr2=0x7f14d81ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:15.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.574+0000 7f14d6ffd640 1 --2- 172.21.15.106:0/274009395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14d813fef0 0x7f14d81ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.574+0000 7f14d6ffd640 1 -- 172.21.15.106:0/274009395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14d80a9aa0 msgr2=0x7f14d81cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:15.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.574+0000 7f14d6ffd640 1 --2- 172.21.15.106:0/274009395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14d80a9aa0 0x7f14d81cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:15.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.574+0000 7f14d6ffd640 10 monclient: _finish_hunting 0 2024-04-04T21:39:15.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.574+0000 7f14d6ffd640 1 monclient: found mon.noname-b 2024-04-04T21:39:15.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.574+0000 7f14d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:15.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.574+0000 7f14d6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:15.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.574+0000 7f14d6ffd640 1 -- 172.21.15.106:0/274009395 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d81cdb90 con 0x7f14d80b5200 2024-04-04T21:39:15.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.576+0000 7f14d6ffd640 1 --2- 172.21.15.106:0/274009395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14d80b5200 0x7f14d81ce570 secure :-1 s=READY pgs=5538 cs=0 l=1 rev1=1 crypto rx=0x7f14cc00e900 tx=0x7f14cc00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:15.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.576+0000 7f14bffff640 10 client.?.objecter ms_handle_connect 0x7f14d80b5200 2024-04-04T21:39:15.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.576+0000 7f14bffff640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:15.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14bffff640 1 -- 172.21.15.106:0/274009395 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc00ca00 con 0x7f14d80b5200 2024-04-04T21:39:15.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:15.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14bffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:15.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14bffff640 10 monclient: dump: 2024-04-04T21:39:15.582 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:15.582 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:15.582 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.582 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.582 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:15.582 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:15.582 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:15.582 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:15.582 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:15.582 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:15.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14bffff640 10 monclient: _finish_auth 0 2024-04-04T21:39:15.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14bffff640 10 monclient: _check_auth_tickets 2024-04-04T21:39:15.582 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14bffff640 1 -- 172.21.15.106:0/274009395 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc00cba0 con 0x7f14d80b5200 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14bffff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14bffff640 1 -- 172.21.15.106:0/274009395 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc004030 con 0x7f14d80b5200 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14ddb93640 5 monclient: authenticate success, global_id 22072 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14bffff640 10 monclient: dump: 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:15.583 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14ddb93640 10 monclient: _renew_subs 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14ddb93640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14d5ffb640 4 set_mon_vals no callback set 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14ddb93640 1 -- 172.21.15.106:0/274009395 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d81ccef0 con 0x7f14d80b5200 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14ddb93640 10 client.22072.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14ddb93640 10 monclient: _renew_subs 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14ddb93640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14ddb93640 1 -- 172.21.15.106:0/274009395 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d81ceff0 con 0x7f14d80b5200 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.580+0000 7f14ddb93640 1 librados: init done 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.581+0000 7f14ddb93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.581+0000 7f14ddb93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.581+0000 7f14ddb93640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:15.584 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.581+0000 7f14ddb93640 1 -- 172.21.15.106:0/274009395 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d804f050 con 0x7f14d80b5200 2024-04-04T21:39:15.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.600+0000 7f14bffff640 1 -- 172.21.15.106:0/274009395 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f14cc0041d0 con 0x7f14d80b5200 2024-04-04T21:39:15.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.600+0000 7f14bffff640 1 --2- 172.21.15.106:0/274009395 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f14c406ceb0 0x7f14c406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:15.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.600+0000 7f14d77fe640 1 --2- 172.21.15.106:0/274009395 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f14c406ceb0 0x7f14c406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:15.600 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.600+0000 7f14d77fe640 10 monclient: get_auth_request con 0x7f14c406ceb0 auth_method 0 2024-04-04T21:39:15.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.601+0000 7f14bffff640 10 client.22072.objecter ms_handle_connect 0x7f14c406ceb0 2024-04-04T21:39:15.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.601+0000 7f14d77fe640 1 --2- 172.21.15.106:0/274009395 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f14c406ceb0 0x7f14c406f370 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f14c8009560 tx=0x7f14c8008030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:15.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.601+0000 7f14bffff640 1 -- 172.21.15.106:0/274009395 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f14cc057190 con 0x7f14d80b5200 2024-04-04T21:39:15.601 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.601+0000 7f14bffff640 10 client.22072.objecter ms_dispatch 0x7f14d8000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:15.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.601+0000 7f14bffff640 3 client.22072.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:15.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.601+0000 7f14bffff640 3 client.22072.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:15.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.602+0000 7f14bffff640 20 client.22072.objecter dump_active .. 0 homeless 2024-04-04T21:39:15.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.841+0000 7f14bffff640 1 -- 172.21.15.106:0/274009395 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f14d81cdb90 con 0x7f14d80b5200 2024-04-04T21:39:15.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.841+0000 7f14bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:15.841 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:15.841+0000 7f14bffff640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:16.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.021+0000 7f14ddb93640 1 -- 172.21.15.106:0/274009395 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f14d81ccba0 con 0x7f14c406ceb0 2024-04-04T21:39:16.022 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.022+0000 7f14bffff640 1 -- 172.21.15.106:0/274009395 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f14d81ccba0 con 0x7f14c406ceb0 2024-04-04T21:39:16.022 INFO:teuthology.orchestra.run.smithi106.stderr:instructed osd(s) 1 to deep-scrub 2024-04-04T21:39:16.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.023+0000 7f14ddb93640 10 librados: watch_flush enter 2024-04-04T21:39:16.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.023+0000 7f14ddb93640 10 librados: watch_flush exit 2024-04-04T21:39:16.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.023+0000 7f14ddb93640 20 client.22072.objecter shutdown clearing up homeless session... 2024-04-04T21:39:16.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.023+0000 7f14ddb93640 10 client.22072.objecter successfully canceled tick 2024-04-04T21:39:16.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 1 -- 172.21.15.106:0/274009395 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f14c406ceb0 msgr2=0x7f14c406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 1 --2- 172.21.15.106:0/274009395 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f14c406ceb0 0x7f14c406f370 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f14c8009560 tx=0x7f14c8008030 comp rx=0 tx=0).stop 2024-04-04T21:39:16.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 10 monclient: shutdown 2024-04-04T21:39:16.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:16.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 1 -- 172.21.15.106:0/274009395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14d80b5200 msgr2=0x7f14d81ce570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 1 --2- 172.21.15.106:0/274009395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14d80b5200 0x7f14d81ce570 secure :-1 s=READY pgs=5538 cs=0 l=1 rev1=1 crypto rx=0x7f14cc00e900 tx=0x7f14cc00edc0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 1 -- 172.21.15.106:0/274009395 shutdown_connections 2024-04-04T21:39:16.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 1 --2- 172.21.15.106:0/274009395 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f14c406ceb0 0x7f14c406f370 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 1 --2- 172.21.15.106:0/274009395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14d813fef0 0x7f14d81ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 1 --2- 172.21.15.106:0/274009395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14d80b5200 0x7f14d81ce570 unknown :-1 s=CLOSED pgs=5538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 1 --2- 172.21.15.106:0/274009395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14d80a9aa0 0x7f14d81cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 1 -- 172.21.15.106:0/274009395 >> 172.21.15.106:0/274009395 conn(0x7f14d8066e40 msgr2=0x7f14d80b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:16.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 1 -- 172.21.15.106:0/274009395 shutdown_connections 2024-04-04T21:39:16.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 1 -- 172.21.15.106:0/274009395 wait complete. 2024-04-04T21:39:16.026 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.024+0000 7f14ddb93640 1 librados: shutdown 2024-04-04T21:39:16.036 INFO:tasks.ceph:Scrubbing osd.2 2024-04-04T21:39:16.036 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-04T21:39:16.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.150+0000 7ff8dcc24640 10 monclient: get_monmap_and_config 2024-04-04T21:39:16.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.150+0000 7ff8dcc24640 10 monclient: build_initial_monmap 2024-04-04T21:39:16.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.150+0000 7ff8dcc24640 10 monclient: monmap: 2024-04-04T21:39:16.151 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:16.151 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:16.151 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:16.151445+0000 2024-04-04T21:39:16.151 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:16.151445+0000 2024-04-04T21:39:16.151 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:16.151 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:16.151 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:16.151 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:16.151 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:16.151 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:16.151 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:16.151 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:16.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 1 Processor -- start 2024-04-04T21:39:16.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 1 -- start start 2024-04-04T21:39:16.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 10 monclient: init 2024-04-04T21:39:16.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:16.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-04T21:39:16.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff8d80b5360 0x7ff8d813f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:16.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 10 monclient(hunting): picked mon.noname-c con 0x7ff8d80b5360 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:16.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff8d813faa0 0x7ff8d8145b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:16.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 10 monclient(hunting): picked mon.noname-a con 0x7ff8d813faa0 addr v2:172.21.15.106:3300/0 2024-04-04T21:39:16.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8d8146090 addr v1:172.21.15.203:6789/0 2024-04-04T21:39:16.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d813ff10 con 0x7ff8d8146090 2024-04-04T21:39:16.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:16.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d80b5740 con 0x7ff8d813faa0 2024-04-04T21:39:16.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:16.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d80a9e70 con 0x7ff8d80b5360 2024-04-04T21:39:16.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:16.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.151+0000 7ff8dcc24640 10 monclient(hunting): authenticate will time out at 3605.419058s 2024-04-04T21:39:16.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.152+0000 7ff8cffff640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff8d813faa0 0x7ff8d8145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:16.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.153+0000 7ff8cffff640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff8d813faa0 0x7ff8d8145b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42436/0 (socket says 172.21.15.106:42436) 2024-04-04T21:39:16.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.153+0000 7ff8cffff640 1 -- 172.21.15.106:0/2381460244 learned_addr learned my addr 172.21.15.106:0/2381460244 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:16.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.153+0000 7ff8cffff640 10 monclient(hunting): get_auth_request con 0x7ff8d813faa0 auth_method 0 2024-04-04T21:39:16.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.153+0000 7ff8cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:16.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.153+0000 7ff8cffff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:16.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.153+0000 7ff8cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:16.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.153+0000 7ff8d659c640 1 --2- 172.21.15.106:0/2381460244 >> v2:172.21.15.203:3300/0 conn(0x7ff8d80b5360 0x7ff8d813f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:16.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.153+0000 7ff8cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:16.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.153+0000 7ff8cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:16.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.153+0000 7ff8cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:16.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.155+0000 7ff8d659c640 10 monclient(hunting): get_auth_request con 0x7ff8d80b5360 auth_method 0 2024-04-04T21:39:16.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.155+0000 7ff8d659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:16.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.155+0000 7ff8d659c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:16.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.155+0000 7ff8d659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:16.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.155+0000 7ff8cffff640 10 monclient(hunting): handle_auth_done global_id 15126 payload 274 2024-04-04T21:39:16.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.155+0000 7ff8cffff640 1 -- 172.21.15.106:0/2381460244 >> v2:172.21.15.203:3300/0 conn(0x7ff8d80b5360 msgr2=0x7ff8d813f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:16.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.155+0000 7ff8cffff640 1 --2- 172.21.15.106:0/2381460244 >> v2:172.21.15.203:3300/0 conn(0x7ff8d80b5360 0x7ff8d813f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.155+0000 7ff8cffff640 1 -- 172.21.15.106:0/2381460244 >> v1:172.21.15.203:6789/0 conn(0x7ff8d8146090 legacy=0x7ff8d80b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:16.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.155+0000 7ff8cffff640 10 monclient: _finish_hunting 0 2024-04-04T21:39:16.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.155+0000 7ff8cffff640 1 monclient: found mon.noname-a 2024-04-04T21:39:16.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.155+0000 7ff8cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:16.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.155+0000 7ff8cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:39:16.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.155+0000 7ff8cffff640 1 -- 172.21.15.106:0/2381460244 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d80ab3e0 con 0x7ff8d813faa0 2024-04-04T21:39:16.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.158+0000 7ff8cffff640 1 --2- 172.21.15.106:0/2381460244 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8d813faa0 0x7ff8d8145b50 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80b5b60 tx=0x7ff8c002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdbe1d49c1f47746 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:16.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8d5d9b640 1 -- 172.21.15.106:0/2381460244 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c002e070 con 0x7ff8d813faa0 2024-04-04T21:39:16.171 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:16.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8d5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:39:16.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8d5d9b640 10 monclient: dump: 2024-04-04T21:39:16.172 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:16.172 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:16.172 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.172 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.172 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:16.172 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:16.172 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:16.172 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:16.172 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:16.172 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:16.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8d5d9b640 10 monclient: _finish_auth 0 2024-04-04T21:39:16.172 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8d5d9b640 10 monclient: _check_auth_tickets 2024-04-04T21:39:16.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8d5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:16.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8d5d9b640 1 -- 172.21.15.106:0/2381460244 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c0037070 con 0x7ff8d813faa0 2024-04-04T21:39:16.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8dcc24640 5 monclient: authenticate success, global_id 15126 2024-04-04T21:39:16.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8dcc24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:39:16.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8d5d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:16.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8dcc24640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:16.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8d5d9b640 1 -- 172.21.15.106:0/2381460244 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0031a00 con 0x7ff8d813faa0 2024-04-04T21:39:16.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8dcc24640 4 set_mon_vals no callback set 2024-04-04T21:39:16.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:16.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8d5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:39:16.173 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.171+0000 7ff8d5d9b640 10 monclient: dump: 2024-04-04T21:39:16.173 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:16.174 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:16.174 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.174 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.174 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:16.174 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:16.174 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:16.174 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:16.174 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:16.174 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:16.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.172+0000 7ff8dcc24640 10 monclient: shutdown 2024-04-04T21:39:16.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.172+0000 7ff8dcc24640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:16.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.172+0000 7ff8dcc24640 1 -- 172.21.15.106:0/2381460244 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8d813faa0 msgr2=0x7ff8d8145b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.172+0000 7ff8dcc24640 1 --2- 172.21.15.106:0/2381460244 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8d813faa0 0x7ff8d8145b50 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80b5b60 tx=0x7ff8c002b880 comp rx=0 tx=0).stop 2024-04-04T21:39:16.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.172+0000 7ff8dcc24640 1 -- 172.21.15.106:0/2381460244 shutdown_connections 2024-04-04T21:39:16.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.172+0000 7ff8dcc24640 1 --2- 172.21.15.106:0/2381460244 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8d813faa0 0x7ff8d8145b50 unknown :-1 s=CLOSED pgs=3162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.172+0000 7ff8dcc24640 1 --2- 172.21.15.106:0/2381460244 >> v2:172.21.15.203:3300/0 conn(0x7ff8d80b5360 0x7ff8d813f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.172+0000 7ff8dcc24640 1 -- 172.21.15.106:0/2381460244 >> 172.21.15.106:0/2381460244 conn(0x7ff8d8066e70 msgr2=0x7ff8d8067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:16.179 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.172+0000 7ff8dcc24640 1 -- 172.21.15.106:0/2381460244 shutdown_connections 2024-04-04T21:39:16.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.172+0000 7ff8dcc24640 1 -- 172.21.15.106:0/2381460244 wait complete. 2024-04-04T21:39:16.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.173+0000 7ff8dcc24640 10 monclient: build_initial_monmap 2024-04-04T21:39:16.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.173+0000 7ff8dcc24640 10 monclient: monmap: 2024-04-04T21:39:16.181 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:16.181 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:16.181 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:16.181 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:16.181 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:16.181 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:16.181 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:16.181 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:16.182 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:16.182 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:16.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.173+0000 7ff8dcc24640 1 librados: starting msgr at 2024-04-04T21:39:16.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.173+0000 7ff8dcc24640 1 librados: starting objecter 2024-04-04T21:39:16.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.173+0000 7ff8dcc24640 1 Processor -- start 2024-04-04T21:39:16.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.173+0000 7ff8dcc24640 1 -- start start 2024-04-04T21:39:16.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.173+0000 7ff8dcc24640 1 librados: setting wanted keys 2024-04-04T21:39:16.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.173+0000 7ff8dcc24640 1 librados: calling monclient init 2024-04-04T21:39:16.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.173+0000 7ff8dcc24640 10 monclient: init 2024-04-04T21:39:16.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:16.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-04T21:39:16.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8d80b5360 0x7ff8d81cf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:16.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 10 monclient(hunting): picked mon.noname-a con 0x7ff8d80b5360 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:16.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8d813faa0 0x7ff8d8139160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:16.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 10 monclient(hunting): picked mon.noname-b con 0x7ff8d813faa0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:16.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d8146090 0x7ff8d81396a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:16.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 10 monclient(hunting): picked mon.noname-c con 0x7ff8d8146090 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:16.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:16.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d8140110 con 0x7ff8d80b5360 2024-04-04T21:39:16.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:16.183 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d80aa300 con 0x7ff8d813faa0 2024-04-04T21:39:16.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:16.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d8146500 con 0x7ff8d8146090 2024-04-04T21:39:16.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:16.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.174+0000 7ff8dcc24640 10 monclient(hunting): authenticate will time out at 3605.441748s 2024-04-04T21:39:16.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.175+0000 7ff8cffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8d813faa0 0x7ff8d8139160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:16.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.175+0000 7ff8cffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8d813faa0 0x7ff8d8139160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50256/0 (socket says 172.21.15.106:50256) 2024-04-04T21:39:16.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.175+0000 7ff8cffff640 1 -- 172.21.15.106:0/1445353552 learned_addr learned my addr 172.21.15.106:0/1445353552 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:16.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.175+0000 7ff8cffff640 10 monclient(hunting): get_auth_request con 0x7ff8d813faa0 auth_method 0 2024-04-04T21:39:16.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.175+0000 7ff8cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:16.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.175+0000 7ff8cffff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:16.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.175+0000 7ff8cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:16.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.176+0000 7ff8d6d9d640 1 --2- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d8146090 0x7ff8d81396a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:16.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.176+0000 7ff8d6d9d640 10 monclient(hunting): get_auth_request con 0x7ff8d8146090 auth_method 0 2024-04-04T21:39:16.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.176+0000 7ff8d6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:16.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.176+0000 7ff8d6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:16.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.176+0000 7ff8d6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:16.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.177+0000 7ff8d6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:16.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.177+0000 7ff8d6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:16.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.177+0000 7ff8d6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:16.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.180+0000 7ff8d6d9d640 10 monclient(hunting): handle_auth_done global_id 17660 payload 824 2024-04-04T21:39:16.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.180+0000 7ff8d6d9d640 1 -- 172.21.15.106:0/1445353552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8d813faa0 msgr2=0x7ff8d8139160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.180+0000 7ff8d6d9d640 1 --2- 172.21.15.106:0/1445353552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8d813faa0 0x7ff8d8139160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.180+0000 7ff8d6d9d640 1 -- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8d80b5360 msgr2=0x7ff8d81cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.180+0000 7ff8d6d9d640 1 --2- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8d80b5360 0x7ff8d81cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.180+0000 7ff8d6d9d640 10 monclient: _finish_hunting 0 2024-04-04T21:39:16.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.180+0000 7ff8d6d9d640 1 monclient: found mon.noname-c 2024-04-04T21:39:16.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.180+0000 7ff8d6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:16.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.180+0000 7ff8d6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:39:16.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.180+0000 7ff8d6d9d640 1 -- 172.21.15.106:0/1445353552 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d8143170 con 0x7ff8d8146090 2024-04-04T21:39:16.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.182+0000 7ff8d6d9d640 1 --2- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d8146090 0x7ff8d81396a0 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000c500 tx=0x7ff8d000c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:16.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.182+0000 7ff8ceffd640 10 client.?.objecter ms_handle_connect 0x7ff8d8146090 2024-04-04T21:39:16.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.182+0000 7ff8ceffd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:16.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8ceffd640 1 -- 172.21.15.106:0/1445353552 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0016020 con 0x7ff8d8146090 2024-04-04T21:39:16.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:16.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8ceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:39:16.228 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8ceffd640 10 monclient: dump: 2024-04-04T21:39:16.228 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8ceffd640 10 monclient: _finish_auth 0 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8ceffd640 10 monclient: _check_auth_tickets 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8dcc24640 5 monclient: authenticate success, global_id 17660 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8dcc24640 10 monclient: _renew_subs 2024-04-04T21:39:16.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8dcc24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:39:16.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8dcc24640 1 -- 172.21.15.106:0/1445353552 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d81ced60 con 0x7ff8d8146090 2024-04-04T21:39:16.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8dcc24640 10 client.17660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:16.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8dcc24640 10 monclient: _renew_subs 2024-04-04T21:39:16.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8dcc24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:39:16.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8dcc24640 1 -- 172.21.15.106:0/1445353552 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d81cf6c0 con 0x7ff8d8146090 2024-04-04T21:39:16.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.228+0000 7ff8dcc24640 1 librados: init done 2024-04-04T21:39:16.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.229+0000 7ff8dcc24640 10 client.17660.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:16.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.229+0000 7ff8dcc24640 15 client.17660.objecter _session_command_op_assign -1 1 2024-04-04T21:39:16.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.229+0000 7ff8dcc24640 20 client.17660.objecter _get_session osd=-1 returning homeless 2024-04-04T21:39:16.231 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.229+0000 7ff8dcc24640 10 client.17660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:16.231 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.229+0000 7ff8dcc24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:39:16.231 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.229+0000 7ff8dcc24640 1 -- 172.21.15.106:0/1445353552 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff8d8069c40 con 0x7ff8d8146090 2024-04-04T21:39:16.231 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.231+0000 7ff8ceffd640 1 -- 172.21.15.106:0/1445353552 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d8069c40 con 0x7ff8d8146090 2024-04-04T21:39:16.231 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.231+0000 7ff8ceffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:16.231 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.231+0000 7ff8ceffd640 1 -- 172.21.15.106:0/1445353552 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0021060 con 0x7ff8d8146090 2024-04-04T21:39:16.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.231+0000 7ff8ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:16.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.231+0000 7ff8ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:39:16.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.231+0000 7ff8ceffd640 10 monclient: dump: 2024-04-04T21:39:16.232 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:16.232 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:16.232 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.232 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.232 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:16.232 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:16.232 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:16.232 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:16.232 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:16.232 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:16.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.231+0000 7ff8d559a640 4 set_mon_vals no callback set 2024-04-04T21:39:16.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.251+0000 7ff8ceffd640 1 -- 172.21.15.106:0/1445353552 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff8d0003970 con 0x7ff8d8146090 2024-04-04T21:39:16.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.252+0000 7ff8ceffd640 1 --2- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff8b806ce60 0x7ff8b806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:16.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.252+0000 7ff8d659c640 1 --2- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff8b806ce60 0x7ff8b806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:16.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.252+0000 7ff8d659c640 10 monclient: get_auth_request con 0x7ff8b806ce60 auth_method 0 2024-04-04T21:39:16.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.252+0000 7ff8ceffd640 10 client.17660.objecter ms_handle_connect 0x7ff8b806ce60 2024-04-04T21:39:16.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.252+0000 7ff8d659c640 1 --2- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff8b806ce60 0x7ff8b806f320 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80098f0 tx=0x7ff8c8008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:16.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.253+0000 7ff8ceffd640 1 -- 172.21.15.106:0/1445353552 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7ff8d0056990 con 0x7ff8d8146090 2024-04-04T21:39:16.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.253+0000 7ff8ceffd640 10 client.17660.objecter ms_dispatch 0x7ff8d8000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:16.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.253+0000 7ff8ceffd640 3 client.17660.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:16.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.253+0000 7ff8ceffd640 3 client.17660.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:16.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8ceffd640 10 client.17660.objecter checking command 1 2024-04-04T21:39:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8ceffd640 1 --2- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:6810/459340769,v1:172.21.15.106:6811/459340769] conn(0x7ff8b8074b70 0x7ff8b8077010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8ceffd640 20 client.17660.objecter _get_session s=0x7ff8b80744a0 osd=2 3 2024-04-04T21:39:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8ceffd640 20 client.17660.objecter put_session s=0x7ff8b80744a0 osd=2 3 2024-04-04T21:39:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8ceffd640 15 client.17660.objecter _session_command_op_remove -1 1 2024-04-04T21:39:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8ceffd640 20 client.17660.objecter _get_session s=0x7ff8b80744a0 osd=2 3 2024-04-04T21:39:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8ceffd640 20 client.17660.objecter get_session s=0x7ff8b80744a0 osd=2 3 2024-04-04T21:39:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8ceffd640 15 client.17660.objecter _session_command_op_assign 2 1 2024-04-04T21:39:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8ceffd640 20 client.17660.objecter put_session s=0x7ff8b80744a0 osd=2 4 2024-04-04T21:39:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8ceffd640 10 client.17660.objecter _send_command 1 2024-04-04T21:39:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8ceffd640 1 -- 172.21.15.106:0/1445353552 --> [v2:172.21.15.106:6810/459340769,v1:172.21.15.106:6811/459340769] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8d0012e50 con 0x7ff8b8074b70 2024-04-04T21:39:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8ceffd640 20 client.17660.objecter dump_active .. 0 homeless 2024-04-04T21:39:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8ceffd640 1 -- 172.21.15.106:0/1445353552 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_get_version_reply(handle=1 version=1197) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8d00586c0 con 0x7ff8d8146090 2024-04-04T21:39:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8ceffd640 10 monclient: handle_get_version_reply finishing 1 version 1197 2024-04-04T21:39:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8cffff640 1 --2- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:6810/459340769,v1:172.21.15.106:6811/459340769] conn(0x7ff8b8074b70 0x7ff8b8077010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:16.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.254+0000 7ff8cffff640 10 monclient: get_auth_request con 0x7ff8b8074b70 auth_method 0 2024-04-04T21:39:16.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.257+0000 7ff8cffff640 1 --2- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:6810/459340769,v1:172.21.15.106:6811/459340769] conn(0x7ff8b8074b70 0x7ff8b8077010 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:16.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.257+0000 7ff8ceffd640 10 client.17660.objecter ms_handle_connect 0x7ff8b8074b70 2024-04-04T21:39:16.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.281+0000 7ff8ceffd640 1 -- 172.21.15.106:0/1445353552 <== osd.2 v2:172.21.15.106:6810/459340769 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff8d0012e50 con 0x7ff8b8074b70 2024-04-04T21:39:16.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.281+0000 7ff8ceffd640 10 client.17660.objecter ms_dispatch 0x7ff8d8000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:39:16.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.281+0000 7ff8ceffd640 10 client.17660.objecter _finish_command 1 = system:0 2024-04-04T21:39:16.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.281+0000 7ff8ceffd640 20 client.17660.objecter put_session s=0x7ff8b80744a0 osd=2 4 2024-04-04T21:39:16.281 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.281+0000 7ff8ceffd640 15 client.17660.objecter _session_command_op_remove 2 1 2024-04-04T21:39:16.298 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.297+0000 7ff8dcc24640 10 client.17660.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-04T21:39:16.298 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.297+0000 7ff8dcc24640 15 client.17660.objecter _session_command_op_assign -1 2 2024-04-04T21:39:16.298 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.297+0000 7ff8dcc24640 20 client.17660.objecter _get_session s=0x7ff8b80744a0 osd=2 3 2024-04-04T21:39:16.298 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.297+0000 7ff8dcc24640 20 client.17660.objecter put_session s=0x7ff8b80744a0 osd=2 3 2024-04-04T21:39:16.298 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.297+0000 7ff8dcc24640 20 client.17660.objecter _get_session s=0x7ff8b80744a0 osd=2 3 2024-04-04T21:39:16.298 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.297+0000 7ff8dcc24640 15 client.17660.objecter _session_command_op_remove -1 2 2024-04-04T21:39:16.298 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.297+0000 7ff8dcc24640 20 client.17660.objecter get_session s=0x7ff8b80744a0 osd=2 3 2024-04-04T21:39:16.298 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.297+0000 7ff8dcc24640 15 client.17660.objecter _session_command_op_assign 2 2 2024-04-04T21:39:16.298 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.297+0000 7ff8dcc24640 20 client.17660.objecter put_session s=0x7ff8b80744a0 osd=2 4 2024-04-04T21:39:16.299 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.297+0000 7ff8dcc24640 10 client.17660.objecter _send_command 2 2024-04-04T21:39:16.299 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.297+0000 7ff8dcc24640 1 -- 172.21.15.106:0/1445353552 --> [v2:172.21.15.106:6810/459340769,v1:172.21.15.106:6811/459340769] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff8d8143670 con 0x7ff8b8074b70 2024-04-04T21:39:16.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.313+0000 7ff8ceffd640 1 -- 172.21.15.106:0/1445353552 <== osd.2 v2:172.21.15.106:6810/459340769 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff8d8143670 con 0x7ff8b8074b70 2024-04-04T21:39:16.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8ceffd640 10 client.17660.objecter ms_dispatch 0x7ff8d8000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:39:16.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8ceffd640 10 client.17660.objecter _finish_command 2 = system:0 2024-04-04T21:39:16.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8ceffd640 20 client.17660.objecter put_session s=0x7ff8b80744a0 osd=2 4 2024-04-04T21:39:16.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8ceffd640 15 client.17660.objecter _session_command_op_remove 2 2 2024-04-04T21:39:16.315 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-04T21:39:16.315 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-04T21:39:16.316 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-04T21:39:16.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 10 librados: watch_flush enter 2024-04-04T21:39:16.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 10 librados: watch_flush exit 2024-04-04T21:39:16.318 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 10 client.17660.objecter close_session for osd.2 2024-04-04T21:39:16.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 1 -- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:6810/459340769,v1:172.21.15.106:6811/459340769] conn(0x7ff8b8074b70 msgr2=0x7ff8b8077010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 1 --2- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:6810/459340769,v1:172.21.15.106:6811/459340769] conn(0x7ff8b8074b70 0x7ff8b8077010 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 20 client.17660.objecter put_session s=0x7ff8b80744a0 osd=2 1 2024-04-04T21:39:16.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 20 client.17660.objecter shutdown clearing up homeless session... 2024-04-04T21:39:16.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 10 client.17660.objecter successfully canceled tick 2024-04-04T21:39:16.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 1 -- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff8b806ce60 msgr2=0x7ff8b806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 1 --2- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7ff8b806ce60 0x7ff8b806f320 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80098f0 tx=0x7ff8c8008040 comp rx=0 tx=0).stop 2024-04-04T21:39:16.319 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 10 monclient: shutdown 2024-04-04T21:39:16.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:16.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 1 -- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d8146090 msgr2=0x7ff8d81396a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 1 --2- 172.21.15.106:0/1445353552 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d8146090 0x7ff8d81396a0 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000c500 tx=0x7ff8d000c9c0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8d6d9d640 1 -- 172.21.15.106:0/1445353552 reap_dead start 2024-04-04T21:39:16.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 1 -- 172.21.15.106:0/1445353552 shutdown_connections 2024-04-04T21:39:16.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 1 -- 172.21.15.106:0/1445353552 >> 172.21.15.106:0/1445353552 conn(0x7ff8d8066e70 msgr2=0x7ff8d813e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:16.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 1 -- 172.21.15.106:0/1445353552 shutdown_connections 2024-04-04T21:39:16.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 1 -- 172.21.15.106:0/1445353552 wait complete. 2024-04-04T21:39:16.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.314+0000 7ff8dcc24640 1 librados: shutdown 2024-04-04T21:39:16.327 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-04T21:39:16.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.427+0000 7f973d61d640 10 monclient: get_monmap_and_config 2024-04-04T21:39:16.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.427+0000 7f973d61d640 10 monclient: build_initial_monmap 2024-04-04T21:39:16.428 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.427+0000 7f973d61d640 10 monclient: monmap: 2024-04-04T21:39:16.428 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:16.428 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:16.428 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:16.428926+0000 2024-04-04T21:39:16.428 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:16.428926+0000 2024-04-04T21:39:16.428 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:16.429 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:16.429 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:16.429 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:16.429 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:16.429 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:16.429 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:16.429 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:16.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.428+0000 7f973d61d640 1 Processor -- start 2024-04-04T21:39:16.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.428+0000 7f973d61d640 1 -- start start 2024-04-04T21:39:16.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.428+0000 7f973d61d640 10 monclient: init 2024-04-04T21:39:16.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.428+0000 7f973d61d640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:16.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.428+0000 7f973d61d640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-04-04T21:39:16.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.429+0000 7f973d61d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97380a9a50 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:16.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.429+0000 7f973d61d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97380a9e30 addr v1:172.21.15.203:6789/0 2024-04-04T21:39:16.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.429+0000 7f973d61d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9738142f90 0x7f973814df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:16.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.429+0000 7f973d61d640 10 monclient(hunting): picked mon.noname-c con 0x7f9738142f90 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:16.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.429+0000 7f973d61d640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97380ab750 con 0x7f97380a9a50 2024-04-04T21:39:16.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.429+0000 7f973d61d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9738143400 con 0x7f97380a9e30 2024-04-04T21:39:16.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.429+0000 7f973d61d640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:16.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.429+0000 7f973d61d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9738140120 con 0x7f9738142f90 2024-04-04T21:39:16.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.429+0000 7f973d61d640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:16.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.429+0000 7f973d61d640 10 monclient(hunting): authenticate will time out at 3605.696261s 2024-04-04T21:39:16.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.429+0000 7f9736ffd640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f97380a9a50 0x7f973813f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:43838/0 (socket says 172.21.15.106:43838) 2024-04-04T21:39:16.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.429+0000 7f9736ffd640 1 -- 172.21.15.106:0/3808490834 learned_addr learned my addr 172.21.15.106:0/3808490834 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:39:16.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.429+0000 7f97377fe640 1 --2- 172.21.15.106:0/3808490834 >> v2:172.21.15.203:3300/0 conn(0x7f9738142f90 0x7f973814df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:16.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.430+0000 7f97377fe640 10 monclient(hunting): get_auth_request con 0x7f9738142f90 auth_method 0 2024-04-04T21:39:16.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.430+0000 7f97377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:16.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.430+0000 7f97377fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:16.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.430+0000 7f97377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:16.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.431+0000 7f97377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:16.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.431+0000 7f97377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:16.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.431+0000 7f97377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:16.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.432+0000 7f9735ffb640 1 -- 172.21.15.106:0/3808490834 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893001875 0 0) 0x7f97380ab750 con 0x7f97380a9a50 2024-04-04T21:39:16.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.432+0000 7f9735ffb640 10 monclient(hunting): _negotiate 2024-04-04T21:39:16.436 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.432+0000 7f9735ffb640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:16.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.432+0000 7f9735ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:16.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.432+0000 7f9735ffb640 10 monclient(hunting): my global_id is 15129 2024-04-04T21:39:16.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.432+0000 7f9735ffb640 1 -- 172.21.15.106:0/3808490834 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9720003520 con 0x7f97380a9a50 2024-04-04T21:39:16.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.433+0000 7f97377fe640 10 monclient(hunting): handle_auth_done global_id 22081 payload 274 2024-04-04T21:39:16.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.433+0000 7f97377fe640 1 -- 172.21.15.106:0/3808490834 >> v1:172.21.15.203:6789/0 conn(0x7f97380a9e30 legacy=0x7f9738142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.433+0000 7f97377fe640 1 -- 172.21.15.106:0/3808490834 >> v1:172.21.15.106:6789/0 conn(0x7f97380a9a50 legacy=0x7f973813f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.433+0000 7f97377fe640 10 monclient: _finish_hunting 0 2024-04-04T21:39:16.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.433+0000 7f97377fe640 1 monclient: found mon.noname-c 2024-04-04T21:39:16.437 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.433+0000 7f97377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:16.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.433+0000 7f97377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:39:16.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.433+0000 7f97377fe640 1 -- 172.21.15.106:0/3808490834 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97380aa660 con 0x7f9738142f90 2024-04-04T21:39:16.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.436+0000 7f97377fe640 1 --2- 172.21.15.106:0/3808490834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9738142f90 0x7f973814df00 secure :-1 s=READY pgs=5541 cs=0 l=1 rev1=1 crypto rx=0x7f9730009d90 tx=0x7f973002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8be8671e5e1fda5 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:16.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f9735ffb640 1 -- 172.21.15.106:0/3808490834 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730039070 con 0x7f9738142f90 2024-04-04T21:39:16.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f9735ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:16.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f9735ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:39:16.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f9735ffb640 10 monclient: dump: 2024-04-04T21:39:16.441 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:16.441 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:16.441 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.442 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.442 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:16.442 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:16.442 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:16.442 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:16.442 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:16.442 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:16.442 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f9735ffb640 10 monclient: _finish_auth 0 2024-04-04T21:39:16.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f9735ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:39:16.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f9735ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:16.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f9735ffb640 1 -- 172.21.15.106:0/3808490834 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973002cc60 con 0x7f9738142f90 2024-04-04T21:39:16.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f9735ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:16.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f9735ffb640 1 -- 172.21.15.106:0/3808490834 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730034660 con 0x7f9738142f90 2024-04-04T21:39:16.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f973d61d640 5 monclient: authenticate success, global_id 22081 2024-04-04T21:39:16.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f973d61d640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:16.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f973d61d640 4 set_mon_vals no callback set 2024-04-04T21:39:16.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f9735ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:16.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f9735ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:16.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.440+0000 7f9735ffb640 10 monclient: dump: 2024-04-04T21:39:16.443 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:16.444 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:16.444 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.444 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.444 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:16.444 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:16.444 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:16.444 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:16.444 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:16.444 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:16.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.441+0000 7f973d61d640 10 monclient: shutdown 2024-04-04T21:39:16.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.441+0000 7f973d61d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:16.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.441+0000 7f973d61d640 1 -- 172.21.15.106:0/3808490834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9738142f90 msgr2=0x7f973814df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.441+0000 7f973d61d640 1 --2- 172.21.15.106:0/3808490834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9738142f90 0x7f973814df00 secure :-1 s=READY pgs=5541 cs=0 l=1 rev1=1 crypto rx=0x7f9730009d90 tx=0x7f973002beb0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.441+0000 7f973d61d640 1 -- 172.21.15.106:0/3808490834 shutdown_connections 2024-04-04T21:39:16.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.441+0000 7f973d61d640 1 --2- 172.21.15.106:0/3808490834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9738142f90 0x7f973814df00 unknown :-1 s=CLOSED pgs=5541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.441+0000 7f973d61d640 1 -- 172.21.15.106:0/3808490834 >> 172.21.15.106:0/3808490834 conn(0x7f97380b8030 msgr2=0x7f97380b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:16.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.441+0000 7f973d61d640 1 -- 172.21.15.106:0/3808490834 shutdown_connections 2024-04-04T21:39:16.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.441+0000 7f973d61d640 1 -- 172.21.15.106:0/3808490834 wait complete. 2024-04-04T21:39:16.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.442+0000 7f973d61d640 10 monclient: build_initial_monmap 2024-04-04T21:39:16.448 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.442+0000 7f973d61d640 10 monclient: monmap: 2024-04-04T21:39:16.448 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:16.448 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:16.448 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:16.448 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:16.448 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:16.448 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:16.448 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:16.448 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:16.448 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.443+0000 7f973d61d640 1 librados: starting msgr at 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.443+0000 7f973d61d640 1 librados: starting objecter 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.443+0000 7f973d61d640 1 Processor -- start 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.443+0000 7f973d61d640 1 -- start start 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.443+0000 7f973d61d640 1 librados: setting wanted keys 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.443+0000 7f973d61d640 1 librados: calling monclient init 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.443+0000 7f973d61d640 10 monclient: init 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97380a9a50 0x7f9738141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 10 monclient(hunting): picked mon.noname-a con 0x7f97380a9a50 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97380a9e30 0x7f9738146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 10 monclient(hunting): picked mon.noname-b con 0x7f97380a9e30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:16.449 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9738142f90 0x7f97381d7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:16.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 10 monclient(hunting): picked mon.noname-c con 0x7f9738142f90 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:16.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:16.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973813ff30 con 0x7f97380a9a50 2024-04-04T21:39:16.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:16.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973813fdb0 con 0x7f97380a9e30 2024-04-04T21:39:16.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:16.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f973814e440 con 0x7f9738142f90 2024-04-04T21:39:16.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:16.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f973d61d640 10 monclient(hunting): authenticate will time out at 3605.711633s 2024-04-04T21:39:16.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.444+0000 7f97377fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9738142f90 0x7f97381d7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:16.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97367fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97380a9e30 0x7f9738146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:16.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97377fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9738142f90 0x7f97381d7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:38124/0 (socket says 172.21.15.106:38124) 2024-04-04T21:39:16.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97377fe640 1 -- 172.21.15.106:0/788815790 learned_addr learned my addr 172.21.15.106:0/788815790 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97377fe640 10 monclient(hunting): get_auth_request con 0x7f9738142f90 auth_method 0 2024-04-04T21:39:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97377fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97367fc640 10 monclient(hunting): get_auth_request con 0x7f97380a9e30 auth_method 0 2024-04-04T21:39:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97367fc640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97367fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.445+0000 7f97367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:16.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.448+0000 7f97367fc640 10 monclient(hunting): handle_auth_done global_id 22084 payload 824 2024-04-04T21:39:16.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.448+0000 7f97367fc640 1 -- 172.21.15.106:0/788815790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9738142f90 msgr2=0x7f97381d7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.448+0000 7f97367fc640 1 --2- 172.21.15.106:0/788815790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9738142f90 0x7f97381d7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.448+0000 7f97367fc640 1 -- 172.21.15.106:0/788815790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97380a9a50 msgr2=0x7f9738141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.448+0000 7f97367fc640 1 --2- 172.21.15.106:0/788815790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97380a9a50 0x7f9738141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.448+0000 7f97367fc640 10 monclient: _finish_hunting 0 2024-04-04T21:39:16.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.448+0000 7f97367fc640 1 monclient: found mon.noname-b 2024-04-04T21:39:16.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.448+0000 7f97367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:16.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.448+0000 7f97367fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:16.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.448+0000 7f97367fc640 1 -- 172.21.15.106:0/788815790 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9738139500 con 0x7f97380a9e30 2024-04-04T21:39:16.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.450+0000 7f97367fc640 1 --2- 172.21.15.106:0/788815790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97380a9e30 0x7f9738146470 secure :-1 s=READY pgs=5542 cs=0 l=1 rev1=1 crypto rx=0x7f972800c340 tx=0x7f972800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:16.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.450+0000 7f971f7fe640 10 client.?.objecter ms_handle_connect 0x7f97380a9e30 2024-04-04T21:39:16.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.450+0000 7f971f7fe640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:16.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f971f7fe640 1 -- 172.21.15.106:0/788815790 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97280052d0 con 0x7f97380a9e30 2024-04-04T21:39:16.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f971f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:16.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f971f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:16.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f971f7fe640 10 monclient: dump: 2024-04-04T21:39:16.455 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:16.455 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:16.455 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.455 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.455 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:16.455 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:16.455 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:16.455 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:16.455 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:16.455 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:16.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f971f7fe640 10 monclient: _finish_auth 0 2024-04-04T21:39:16.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f971f7fe640 10 monclient: _check_auth_tickets 2024-04-04T21:39:16.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f971f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f971f7fe640 1 -- 172.21.15.106:0/788815790 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9728005490 con 0x7f97380a9e30 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f971f7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f971f7fe640 1 -- 172.21.15.106:0/788815790 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9728007260 con 0x7f97380a9e30 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f973d61d640 5 monclient: authenticate success, global_id 22084 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f971f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f971f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f971f7fe640 10 monclient: dump: 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:16.456 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:16.457 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:16.457 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:16.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f973d61d640 10 monclient: _renew_subs 2024-04-04T21:39:16.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f973d61d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:16.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f973d61d640 1 -- 172.21.15.106:0/788815790 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97381d7cb0 con 0x7f97380a9e30 2024-04-04T21:39:16.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f9734ff9640 4 set_mon_vals no callback set 2024-04-04T21:39:16.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f973d61d640 10 client.22084.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:16.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f973d61d640 10 monclient: _renew_subs 2024-04-04T21:39:16.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f973d61d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:16.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f973d61d640 1 -- 172.21.15.106:0/788815790 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f973813c260 con 0x7f97380a9e30 2024-04-04T21:39:16.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.454+0000 7f973d61d640 1 librados: init done 2024-04-04T21:39:16.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.455+0000 7f973d61d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:16.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.455+0000 7f973d61d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:16.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.455+0000 7f973d61d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:16.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.455+0000 7f973d61d640 1 -- 172.21.15.106:0/788815790 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f973804f050 con 0x7f97380a9e30 2024-04-04T21:39:16.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.477+0000 7f971f7fe640 1 -- 172.21.15.106:0/788815790 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9728007400 con 0x7f97380a9e30 2024-04-04T21:39:16.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.477+0000 7f971f7fe640 1 --2- 172.21.15.106:0/788815790 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f972406ceb0 0x7f972406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:16.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.477+0000 7f9736ffd640 1 --2- 172.21.15.106:0/788815790 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f972406ceb0 0x7f972406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:16.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.477+0000 7f9736ffd640 10 monclient: get_auth_request con 0x7f972406ceb0 auth_method 0 2024-04-04T21:39:16.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.478+0000 7f9736ffd640 1 --2- 172.21.15.106:0/788815790 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f972406ceb0 0x7f972406f370 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f972c000f60 tx=0x7f972c0078f0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:16.478 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.478+0000 7f971f7fe640 10 client.22084.objecter ms_handle_connect 0x7f972406ceb0 2024-04-04T21:39:16.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.479+0000 7f971f7fe640 1 -- 172.21.15.106:0/788815790 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f9728056380 con 0x7f97380a9e30 2024-04-04T21:39:16.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.479+0000 7f971f7fe640 10 client.22084.objecter ms_dispatch 0x7f9738000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:16.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.479+0000 7f971f7fe640 3 client.22084.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:16.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.479+0000 7f971f7fe640 3 client.22084.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:16.479 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.479+0000 7f971f7fe640 20 client.22084.objecter dump_active .. 0 homeless 2024-04-04T21:39:16.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.719+0000 7f971f7fe640 1 -- 172.21.15.106:0/788815790 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f97280582e0 con 0x7f97380a9e30 2024-04-04T21:39:16.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.719+0000 7f971f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:16.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.719+0000 7f971f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:16.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.897+0000 7f973d61d640 1 -- 172.21.15.106:0/788815790 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f97381469b0 con 0x7f972406ceb0 2024-04-04T21:39:16.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.898+0000 7f971f7fe640 1 -- 172.21.15.106:0/788815790 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f972c005890 con 0x7f972406ceb0 2024-04-04T21:39:16.898 INFO:teuthology.orchestra.run.smithi106.stderr:instructed osd(s) 2 to deep-scrub 2024-04-04T21:39:16.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 10 librados: watch_flush enter 2024-04-04T21:39:16.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 10 librados: watch_flush exit 2024-04-04T21:39:16.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 20 client.22084.objecter shutdown clearing up homeless session... 2024-04-04T21:39:16.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 10 client.22084.objecter successfully canceled tick 2024-04-04T21:39:16.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 1 -- 172.21.15.106:0/788815790 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f972406ceb0 msgr2=0x7f972406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 1 --2- 172.21.15.106:0/788815790 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f972406ceb0 0x7f972406f370 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f972c000f60 tx=0x7f972c0078f0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 10 monclient: shutdown 2024-04-04T21:39:16.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:16.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 1 -- 172.21.15.106:0/788815790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97380a9e30 msgr2=0x7f9738146470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:16.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 1 --2- 172.21.15.106:0/788815790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97380a9e30 0x7f9738146470 secure :-1 s=READY pgs=5542 cs=0 l=1 rev1=1 crypto rx=0x7f972800c340 tx=0x7f972800c800 comp rx=0 tx=0).stop 2024-04-04T21:39:16.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 1 -- 172.21.15.106:0/788815790 shutdown_connections 2024-04-04T21:39:16.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 1 --2- 172.21.15.106:0/788815790 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f972406ceb0 0x7f972406f370 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 1 --2- 172.21.15.106:0/788815790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9738142f90 0x7f97381d7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 1 --2- 172.21.15.106:0/788815790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97380a9e30 0x7f9738146470 unknown :-1 s=CLOSED pgs=5542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 1 --2- 172.21.15.106:0/788815790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97380a9a50 0x7f9738141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:16.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 1 -- 172.21.15.106:0/788815790 >> 172.21.15.106:0/788815790 conn(0x7f97380b8030 msgr2=0x7f973813f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:16.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 1 -- 172.21.15.106:0/788815790 shutdown_connections 2024-04-04T21:39:16.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 1 -- 172.21.15.106:0/788815790 wait complete. 2024-04-04T21:39:16.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:16.900+0000 7f973d61d640 1 librados: shutdown 2024-04-04T21:39:16.912 INFO:tasks.ceph:Scrubbing osd.3 2024-04-04T21:39:16.912 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-04T21:39:17.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.023+0000 7f4bab769640 10 monclient: get_monmap_and_config 2024-04-04T21:39:17.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.023+0000 7f4bab769640 10 monclient: build_initial_monmap 2024-04-04T21:39:17.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.023+0000 7f4bab769640 10 monclient: monmap: 2024-04-04T21:39:17.024 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:17.024 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.025 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:17.024225+0000 2024-04-04T21:39:17.025 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:17.024225+0000 2024-04-04T21:39:17.025 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:17.025 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.025 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:17.026 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:17.026 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:17.026 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:17.026 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:17.026 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.026 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.023+0000 7f4bab769640 1 Processor -- start 2024-04-04T21:39:17.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.023+0000 7f4bab769640 1 -- start start 2024-04-04T21:39:17.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.023+0000 7f4bab769640 10 monclient: init 2024-04-04T21:39:17.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.024+0000 7f4bab769640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:17.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.024+0000 7f4bab769640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-04-04T21:39:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.024+0000 7f4bab769640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ba40b5360 0x7f4ba413f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.024+0000 7f4bab769640 10 monclient(hunting): picked mon.noname-b con 0x7f4ba40b5360 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.024+0000 7f4bab769640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4ba413faa0 0x7f4ba4145b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.024+0000 7f4bab769640 10 monclient(hunting): picked mon.noname-a con 0x7f4ba413faa0 addr v2:172.21.15.106:3300/0 2024-04-04T21:39:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.024+0000 7f4bab769640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ba4146090 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.024+0000 7f4bab769640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba413fee0 con 0x7f4ba4146090 2024-04-04T21:39:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.024+0000 7f4bab769640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.024+0000 7f4bab769640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba40b5740 con 0x7f4ba413faa0 2024-04-04T21:39:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.024+0000 7f4bab769640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.024+0000 7f4bab769640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba40a9e70 con 0x7f4ba40b5360 2024-04-04T21:39:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.024+0000 7f4bab769640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.024+0000 7f4bab769640 10 monclient(hunting): authenticate will time out at 3606.291743s 2024-04-04T21:39:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.025+0000 7f4ba9505640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ba40b5360 0x7f4ba413f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.025+0000 7f4ba8d04640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4ba413faa0 0x7f4ba4145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.025+0000 7f4ba9505640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ba40b5360 0x7f4ba413f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:38140/0 (socket says 172.21.15.106:38140) 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.025+0000 7f4ba9505640 1 -- 172.21.15.106:0/2012412038 learned_addr learned my addr 172.21.15.106:0/2012412038 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.025+0000 7f4ba9505640 10 monclient(hunting): get_auth_request con 0x7f4ba40b5360 auth_method 0 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.025+0000 7f4ba9505640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.025+0000 7f4ba9505640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.025+0000 7f4ba9505640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.025+0000 7f4ba8d04640 10 monclient(hunting): get_auth_request con 0x7f4ba413faa0 auth_method 0 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.025+0000 7f4ba8d04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.025+0000 7f4ba8d04640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.025+0000 7f4ba8d04640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.026+0000 7f4ba9505640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.026+0000 7f4ba9505640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.026+0000 7f4ba9505640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.026+0000 7f4ba8d04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:17.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.026+0000 7f4ba8d04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:17.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.026+0000 7f4ba8d04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:17.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.028+0000 7f4ba9505640 10 monclient(hunting): handle_auth_done global_id 17666 payload 274 2024-04-04T21:39:17.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.028+0000 7f4ba9505640 1 -- 172.21.15.106:0/2012412038 >> v2:172.21.15.106:3300/0 conn(0x7f4ba413faa0 msgr2=0x7f4ba4145b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:17.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.028+0000 7f4ba9505640 1 --2- 172.21.15.106:0/2012412038 >> v2:172.21.15.106:3300/0 conn(0x7f4ba413faa0 0x7f4ba4145b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.028+0000 7f4ba9505640 1 -- 172.21.15.106:0/2012412038 >> v1:172.21.15.106:6789/0 conn(0x7f4ba4146090 legacy=0x7f4ba40b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:17.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.028+0000 7f4ba9505640 10 monclient: _finish_hunting 0 2024-04-04T21:39:17.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.028+0000 7f4ba9505640 1 monclient: found mon.noname-b 2024-04-04T21:39:17.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.028+0000 7f4ba9505640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:17.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.028+0000 7f4ba9505640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:39:17.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.028+0000 7f4ba9505640 1 -- 172.21.15.106:0/2012412038 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba40ab3e0 con 0x7f4ba40b5360 2024-04-04T21:39:17.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.030+0000 7f4ba9505640 1 --2- 172.21.15.106:0/2012412038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ba40b5360 0x7f4ba413f560 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7f4b94009870 tx=0x7f4b9402b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71f9a37c30d372ff server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:17.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4b9bfff640 1 -- 172.21.15.106:0/2012412038 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b94038070 con 0x7f4ba40b5360 2024-04-04T21:39:17.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4b9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:17.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4b9bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:17.035 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4b9bfff640 10 monclient: dump: 2024-04-04T21:39:17.035 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:17.035 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.035 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.035 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.035 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:17.035 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.036 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:17.036 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:17.036 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:17.036 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4b9bfff640 10 monclient: _finish_auth 0 2024-04-04T21:39:17.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4b9bfff640 10 monclient: _check_auth_tickets 2024-04-04T21:39:17.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4b9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:17.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4b9bfff640 1 -- 172.21.15.106:0/2012412038 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b94031740 con 0x7f4ba40b5360 2024-04-04T21:39:17.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4bab769640 5 monclient: authenticate success, global_id 17666 2024-04-04T21:39:17.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4bab769640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:39:17.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4b9bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:17.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4bab769640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:17.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4bab769640 4 set_mon_vals no callback set 2024-04-04T21:39:17.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4b9bfff640 1 -- 172.21.15.106:0/2012412038 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9402f050 con 0x7f4ba40b5360 2024-04-04T21:39:17.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4b9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:17.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4b9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:39:17.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.034+0000 7f4b9bfff640 10 monclient: dump: 2024-04-04T21:39:17.037 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:17.037 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.037 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.037 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.037 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:17.037 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.038 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:17.038 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:17.038 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:17.038 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.035+0000 7f4bab769640 10 monclient: shutdown 2024-04-04T21:39:17.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.035+0000 7f4bab769640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:17.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.035+0000 7f4bab769640 1 -- 172.21.15.106:0/2012412038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ba40b5360 msgr2=0x7f4ba413f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.035+0000 7f4bab769640 1 --2- 172.21.15.106:0/2012412038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ba40b5360 0x7f4ba413f560 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7f4b94009870 tx=0x7f4b9402b7e0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.035+0000 7f4bab769640 1 -- 172.21.15.106:0/2012412038 shutdown_connections 2024-04-04T21:39:17.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.035+0000 7f4bab769640 1 --2- 172.21.15.106:0/2012412038 >> v2:172.21.15.106:3300/0 conn(0x7f4ba413faa0 0x7f4ba4145b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.035+0000 7f4bab769640 1 --2- 172.21.15.106:0/2012412038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ba40b5360 0x7f4ba413f560 unknown :-1 s=CLOSED pgs=3269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.035+0000 7f4bab769640 1 -- 172.21.15.106:0/2012412038 >> 172.21.15.106:0/2012412038 conn(0x7f4ba4066e70 msgr2=0x7f4ba4067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:17.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.035+0000 7f4bab769640 1 -- 172.21.15.106:0/2012412038 shutdown_connections 2024-04-04T21:39:17.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.035+0000 7f4bab769640 1 -- 172.21.15.106:0/2012412038 wait complete. 2024-04-04T21:39:17.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.035+0000 7f4bab769640 10 monclient: build_initial_monmap 2024-04-04T21:39:17.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.035+0000 7f4bab769640 10 monclient: monmap: 2024-04-04T21:39:17.043 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:17.043 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:17.043 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:17.043 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:17.043 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:17.043 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.043 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:17.043 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:17.043 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:17.044 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 1 librados: starting msgr at 2024-04-04T21:39:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 1 librados: starting objecter 2024-04-04T21:39:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 1 Processor -- start 2024-04-04T21:39:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 1 -- start start 2024-04-04T21:39:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 1 librados: setting wanted keys 2024-04-04T21:39:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 1 librados: calling monclient init 2024-04-04T21:39:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 10 monclient: init 2024-04-04T21:39:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:39:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ba40b5360 0x7f4ba4141f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 10 monclient(hunting): picked mon.noname-c con 0x7f4ba40b5360 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ba413faa0 0x7f4ba41cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 10 monclient(hunting): picked mon.noname-b con 0x7f4ba413faa0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ba4146090 0x7f4ba41cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 10 monclient(hunting): picked mon.noname-a con 0x7f4ba4146090 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba4146520 con 0x7f4ba4146090 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba40aa300 con 0x7f4ba413faa0 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.036+0000 7f4bab769640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba4140110 con 0x7f4ba40b5360 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4bab769640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4bab769640 10 monclient(hunting): authenticate will time out at 3606.304189s 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4ba9505640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ba40b5360 0x7f4ba4141f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4ba9d06640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ba4146090 0x7f4ba41cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4ba8d04640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ba413faa0 0x7f4ba41cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4ba9505640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ba40b5360 0x7f4ba4141f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:38156/0 (socket says 172.21.15.106:38156) 2024-04-04T21:39:17.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4ba9505640 1 -- 172.21.15.106:0/1731087072 learned_addr learned my addr 172.21.15.106:0/1731087072 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:17.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4ba9505640 10 monclient(hunting): get_auth_request con 0x7f4ba40b5360 auth_method 0 2024-04-04T21:39:17.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4ba9505640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:17.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4ba9505640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:17.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4ba9505640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:17.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4ba9d06640 10 monclient(hunting): get_auth_request con 0x7f4ba4146090 auth_method 0 2024-04-04T21:39:17.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4ba9d06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:17.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4ba9d06640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:17.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.037+0000 7f4ba9d06640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:17.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.038+0000 7f4ba8d04640 10 monclient(hunting): get_auth_request con 0x7f4ba413faa0 auth_method 0 2024-04-04T21:39:17.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.038+0000 7f4ba8d04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:17.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.038+0000 7f4ba8d04640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:17.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.038+0000 7f4ba8d04640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:17.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.038+0000 7f4ba9505640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:17.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.038+0000 7f4ba9505640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:17.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.038+0000 7f4ba9505640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:17.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.038+0000 7f4ba9d06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:17.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.038+0000 7f4ba9d06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:17.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.038+0000 7f4ba9d06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:17.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.038+0000 7f4ba8d04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:17.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.038+0000 7f4ba8d04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:17.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.038+0000 7f4ba8d04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:17.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.041+0000 7f4ba8d04640 10 monclient(hunting): handle_auth_done global_id 22087 payload 824 2024-04-04T21:39:17.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.041+0000 7f4ba8d04640 1 -- 172.21.15.106:0/1731087072 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ba40b5360 msgr2=0x7f4ba4141f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.041+0000 7f4ba8d04640 1 --2- 172.21.15.106:0/1731087072 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ba40b5360 0x7f4ba4141f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.041+0000 7f4ba8d04640 1 -- 172.21.15.106:0/1731087072 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ba4146090 msgr2=0x7f4ba41cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.041+0000 7f4ba8d04640 1 --2- 172.21.15.106:0/1731087072 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ba4146090 0x7f4ba41cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.041+0000 7f4ba8d04640 10 monclient: _finish_hunting 0 2024-04-04T21:39:17.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.041+0000 7f4ba8d04640 1 monclient: found mon.noname-b 2024-04-04T21:39:17.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.041+0000 7f4ba8d04640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:17.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.041+0000 7f4ba8d04640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:17.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.041+0000 7f4ba8d04640 1 -- 172.21.15.106:0/1731087072 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba41ce0f0 con 0x7f4ba413faa0 2024-04-04T21:39:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.043+0000 7f4ba8d04640 1 --2- 172.21.15.106:0/1731087072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ba413faa0 0x7f4ba41cdbb0 secure :-1 s=READY pgs=5543 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000e850 tx=0x7f4ba000ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.044+0000 7f4b99ffb640 10 client.?.objecter ms_handle_connect 0x7f4ba413faa0 2024-04-04T21:39:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.044+0000 7f4b99ffb640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4b99ffb640 1 -- 172.21.15.106:0/1731087072 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0017070 con 0x7f4ba413faa0 2024-04-04T21:39:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4b99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4b99ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4b99ffb640 10 monclient: dump: 2024-04-04T21:39:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4b99ffb640 10 monclient: _finish_auth 0 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4b99ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4b99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4b99ffb640 1 -- 172.21.15.106:0/1731087072 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba000c940 con 0x7f4ba413faa0 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4bab769640 5 monclient: authenticate success, global_id 22087 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4bab769640 10 monclient: _renew_subs 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4bab769640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4bab769640 1 -- 172.21.15.106:0/1731087072 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba41cd690 con 0x7f4ba413faa0 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4b99ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4b99ffb640 1 -- 172.21.15.106:0/1731087072 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba000cd10 con 0x7f4ba413faa0 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4b99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4b99ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4b99ffb640 10 monclient: dump: 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4b9b7fe640 4 set_mon_vals no callback set 2024-04-04T21:39:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4bab769640 10 client.22087.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:17.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4bab769640 10 monclient: _renew_subs 2024-04-04T21:39:17.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4bab769640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:17.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4bab769640 1 -- 172.21.15.106:0/1731087072 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba41cf670 con 0x7f4ba413faa0 2024-04-04T21:39:17.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.047+0000 7f4bab769640 1 librados: init done 2024-04-04T21:39:17.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.048+0000 7f4bab769640 10 client.22087.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:17.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.048+0000 7f4bab769640 15 client.22087.objecter _session_command_op_assign -1 1 2024-04-04T21:39:17.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.048+0000 7f4bab769640 20 client.22087.objecter _get_session osd=-1 returning homeless 2024-04-04T21:39:17.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.048+0000 7f4bab769640 10 client.22087.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:17.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.048+0000 7f4bab769640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:17.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.048+0000 7f4bab769640 1 -- 172.21.15.106:0/1731087072 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4ba4069c40 con 0x7f4ba413faa0 2024-04-04T21:39:17.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.067+0000 7f4b99ffb640 1 -- 172.21.15.106:0/1731087072 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4ba001a070 con 0x7f4ba413faa0 2024-04-04T21:39:17.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.068+0000 7f4b99ffb640 1 --2- 172.21.15.106:0/1731087072 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f4b8806ceb0 0x7f4b8806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.068+0000 7f4ba9505640 1 --2- 172.21.15.106:0/1731087072 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f4b8806ceb0 0x7f4b8806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.068+0000 7f4ba9505640 10 monclient: get_auth_request con 0x7f4b8806ceb0 auth_method 0 2024-04-04T21:39:17.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.068+0000 7f4ba9505640 1 --2- 172.21.15.106:0/1731087072 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f4b8806ceb0 0x7f4b8806f370 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4b9402b820 tx=0x7f4b94008030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:17.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.068+0000 7f4b99ffb640 10 client.22087.objecter ms_handle_connect 0x7f4b8806ceb0 2024-04-04T21:39:17.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 1 -- 172.21.15.106:0/1731087072 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f4ba0057120 con 0x7f4ba413faa0 2024-04-04T21:39:17.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 10 client.22087.objecter ms_dispatch 0x7f4ba4000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:17.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 3 client.22087.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:17.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 3 client.22087.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:17.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 10 client.22087.objecter checking command 1 2024-04-04T21:39:17.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 1 --2- 172.21.15.106:0/1731087072 >> [v2:172.21.15.106:6802/873500775,v1:172.21.15.106:6803/873500775] conn(0x7f4b88074c20 0x7f4b880770c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 20 client.22087.objecter _get_session s=0x7f4b88074550 osd=3 3 2024-04-04T21:39:17.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 20 client.22087.objecter put_session s=0x7f4b88074550 osd=3 3 2024-04-04T21:39:17.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 15 client.22087.objecter _session_command_op_remove -1 1 2024-04-04T21:39:17.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 20 client.22087.objecter _get_session s=0x7f4b88074550 osd=3 3 2024-04-04T21:39:17.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 20 client.22087.objecter get_session s=0x7f4b88074550 osd=3 3 2024-04-04T21:39:17.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 15 client.22087.objecter _session_command_op_assign 3 1 2024-04-04T21:39:17.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 20 client.22087.objecter put_session s=0x7f4b88074550 osd=3 4 2024-04-04T21:39:17.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 10 client.22087.objecter _send_command 1 2024-04-04T21:39:17.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 1 -- 172.21.15.106:0/1731087072 --> [v2:172.21.15.106:6802/873500775,v1:172.21.15.106:6803/873500775] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ba0001d30 con 0x7f4b88074c20 2024-04-04T21:39:17.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 20 client.22087.objecter dump_active .. 0 homeless 2024-04-04T21:39:17.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 1 -- 172.21.15.106:0/1731087072 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=1197) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ba0055320 con 0x7f4ba413faa0 2024-04-04T21:39:17.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4b99ffb640 10 monclient: handle_get_version_reply finishing 1 version 1197 2024-04-04T21:39:17.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.070+0000 7f4ba9d06640 1 --2- 172.21.15.106:0/1731087072 >> [v2:172.21.15.106:6802/873500775,v1:172.21.15.106:6803/873500775] conn(0x7f4b88074c20 0x7f4b880770c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.072 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.071+0000 7f4ba9d06640 10 monclient: get_auth_request con 0x7f4b88074c20 auth_method 0 2024-04-04T21:39:17.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.074+0000 7f4ba9d06640 1 --2- 172.21.15.106:0/1731087072 >> [v2:172.21.15.106:6802/873500775,v1:172.21.15.106:6803/873500775] conn(0x7f4b88074c20 0x7f4b880770c0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:17.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.074+0000 7f4b99ffb640 10 client.22087.objecter ms_handle_connect 0x7f4b88074c20 2024-04-04T21:39:17.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.099+0000 7f4b99ffb640 1 -- 172.21.15.106:0/1731087072 <== osd.3 v2:172.21.15.106:6802/873500775 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4ba0001d30 con 0x7f4b88074c20 2024-04-04T21:39:17.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.099+0000 7f4b99ffb640 10 client.22087.objecter ms_dispatch 0x7f4ba4000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:39:17.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.099+0000 7f4b99ffb640 10 client.22087.objecter _finish_command 1 = system:0 2024-04-04T21:39:17.100 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.099+0000 7f4b99ffb640 20 client.22087.objecter put_session s=0x7f4b88074550 osd=3 4 2024-04-04T21:39:17.100 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.099+0000 7f4b99ffb640 15 client.22087.objecter _session_command_op_remove 3 1 2024-04-04T21:39:17.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.113+0000 7f4bab769640 10 client.22087.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-04T21:39:17.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.113+0000 7f4bab769640 15 client.22087.objecter _session_command_op_assign -1 2 2024-04-04T21:39:17.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.113+0000 7f4bab769640 20 client.22087.objecter _get_session s=0x7f4b88074550 osd=3 3 2024-04-04T21:39:17.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.113+0000 7f4bab769640 20 client.22087.objecter put_session s=0x7f4b88074550 osd=3 3 2024-04-04T21:39:17.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.113+0000 7f4bab769640 20 client.22087.objecter _get_session s=0x7f4b88074550 osd=3 3 2024-04-04T21:39:17.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.113+0000 7f4bab769640 15 client.22087.objecter _session_command_op_remove -1 2 2024-04-04T21:39:17.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.113+0000 7f4bab769640 20 client.22087.objecter get_session s=0x7f4b88074550 osd=3 3 2024-04-04T21:39:17.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.113+0000 7f4bab769640 15 client.22087.objecter _session_command_op_assign 3 2 2024-04-04T21:39:17.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.113+0000 7f4bab769640 20 client.22087.objecter put_session s=0x7f4b88074550 osd=3 4 2024-04-04T21:39:17.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.113+0000 7f4bab769640 10 client.22087.objecter _send_command 2 2024-04-04T21:39:17.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.113+0000 7f4bab769640 1 -- 172.21.15.106:0/1731087072 --> [v2:172.21.15.106:6802/873500775,v1:172.21.15.106:6803/873500775] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4ba40b3da0 con 0x7f4b88074c20 2024-04-04T21:39:17.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.126+0000 7f4b99ffb640 1 -- 172.21.15.106:0/1731087072 <== osd.3 v2:172.21.15.106:6802/873500775 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4ba40b3da0 con 0x7f4b88074c20 2024-04-04T21:39:17.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.126+0000 7f4b99ffb640 10 client.22087.objecter ms_dispatch 0x7f4ba4000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:39:17.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.126+0000 7f4b99ffb640 10 client.22087.objecter _finish_command 2 = system:0 2024-04-04T21:39:17.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.126+0000 7f4b99ffb640 20 client.22087.objecter put_session s=0x7f4b88074550 osd=3 4 2024-04-04T21:39:17.127 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.126+0000 7f4b99ffb640 15 client.22087.objecter _session_command_op_remove 3 2 2024-04-04T21:39:17.127 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-04T21:39:17.127 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-04T21:39:17.127 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-04T21:39:17.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 10 librados: watch_flush enter 2024-04-04T21:39:17.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 10 librados: watch_flush exit 2024-04-04T21:39:17.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 10 client.22087.objecter close_session for osd.3 2024-04-04T21:39:17.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 1 -- 172.21.15.106:0/1731087072 >> [v2:172.21.15.106:6802/873500775,v1:172.21.15.106:6803/873500775] conn(0x7f4b88074c20 msgr2=0x7f4b880770c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.128 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 1 --2- 172.21.15.106:0/1731087072 >> [v2:172.21.15.106:6802/873500775,v1:172.21.15.106:6803/873500775] conn(0x7f4b88074c20 0x7f4b880770c0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 20 client.22087.objecter put_session s=0x7f4b88074550 osd=3 1 2024-04-04T21:39:17.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 20 client.22087.objecter shutdown clearing up homeless session... 2024-04-04T21:39:17.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 10 client.22087.objecter successfully canceled tick 2024-04-04T21:39:17.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 1 -- 172.21.15.106:0/1731087072 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f4b8806ceb0 msgr2=0x7f4b8806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 1 --2- 172.21.15.106:0/1731087072 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f4b8806ceb0 0x7f4b8806f370 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4b9402b820 tx=0x7f4b94008030 comp rx=0 tx=0).stop 2024-04-04T21:39:17.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 10 monclient: shutdown 2024-04-04T21:39:17.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:17.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 1 -- 172.21.15.106:0/1731087072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ba413faa0 msgr2=0x7f4ba41cdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 1 --2- 172.21.15.106:0/1731087072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ba413faa0 0x7f4ba41cdbb0 secure :-1 s=READY pgs=5543 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000e850 tx=0x7f4ba000ed10 comp rx=0 tx=0).stop 2024-04-04T21:39:17.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4ba9d06640 1 -- 172.21.15.106:0/1731087072 reap_dead start 2024-04-04T21:39:17.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 1 -- 172.21.15.106:0/1731087072 shutdown_connections 2024-04-04T21:39:17.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 1 -- 172.21.15.106:0/1731087072 >> 172.21.15.106:0/1731087072 conn(0x7f4ba4066e70 msgr2=0x7f4ba413ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:17.130 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 1 -- 172.21.15.106:0/1731087072 shutdown_connections 2024-04-04T21:39:17.130 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 1 -- 172.21.15.106:0/1731087072 wait complete. 2024-04-04T21:39:17.130 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.127+0000 7f4bab769640 1 librados: shutdown 2024-04-04T21:39:17.137 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-04T21:39:17.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.241+0000 7f1e36a9e640 10 monclient: get_monmap_and_config 2024-04-04T21:39:17.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.241+0000 7f1e36a9e640 10 monclient: build_initial_monmap 2024-04-04T21:39:17.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.241+0000 7f1e36a9e640 10 monclient: monmap: 2024-04-04T21:39:17.242 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:17.242 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.242 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:17.242748+0000 2024-04-04T21:39:17.242 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:17.242748+0000 2024-04-04T21:39:17.242 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:17.242 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.242 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:17.242 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:17.242 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:17.242 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:17.242 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:17.242 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.242+0000 7f1e36a9e640 1 Processor -- start 2024-04-04T21:39:17.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.242+0000 7f1e36a9e640 1 -- start start 2024-04-04T21:39:17.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.242+0000 7f1e36a9e640 10 monclient: init 2024-04-04T21:39:17.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.243+0000 7f1e36a9e640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:17.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.243+0000 7f1e36a9e640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-04-04T21:39:17.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.243+0000 7f1e36a9e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e300a9a50 addr v1:172.21.15.203:6789/0 2024-04-04T21:39:17.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.243+0000 7f1e36a9e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1e300a9e30 0x7f1e30142880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.243+0000 7f1e36a9e640 10 monclient(hunting): picked mon.noname-c con 0x7f1e300a9e30 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:17.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.243+0000 7f1e36a9e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e30142dc0 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:17.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.243+0000 7f1e36a9e640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e300ab750 con 0x7f1e30142dc0 2024-04-04T21:39:17.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.243+0000 7f1e36a9e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e30140120 con 0x7f1e300a9a50 2024-04-04T21:39:17.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.243+0000 7f1e36a9e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:17.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.243+0000 7f1e36a9e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e300aa660 con 0x7f1e300a9e30 2024-04-04T21:39:17.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.243+0000 7f1e36a9e640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:17.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.243+0000 7f1e36a9e640 10 monclient(hunting): authenticate will time out at 3606.510369s 2024-04-04T21:39:17.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.243+0000 7f1e2ffff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1e300a9e30 0x7f1e30142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.244+0000 7f1e2ffff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1e300a9e30 0x7f1e30142880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50282/0 (socket says 172.21.15.106:50282) 2024-04-04T21:39:17.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.244+0000 7f1e2ffff640 1 -- 172.21.15.106:0/2350258324 learned_addr learned my addr 172.21.15.106:0/2350258324 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:17.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.244+0000 7f1e2ffff640 10 monclient(hunting): get_auth_request con 0x7f1e300a9e30 auth_method 0 2024-04-04T21:39:17.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.244+0000 7f1e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:17.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.244+0000 7f1e2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:17.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.244+0000 7f1e2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:17.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.244+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:17.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.244+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:17.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.244+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:17.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.246+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_done global_id 22090 payload 274 2024-04-04T21:39:17.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.246+0000 7f1e2ffff640 1 -- 172.21.15.106:0/2350258324 >> v1:172.21.15.203:6789/0 conn(0x7f1e300a9a50 legacy=0x7f1e3013f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.246+0000 7f1e2ffff640 1 -- 172.21.15.106:0/2350258324 >> v1:172.21.15.106:6789/0 conn(0x7f1e30142dc0 legacy=0x7f1e3014cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:17.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.246+0000 7f1e2ffff640 10 monclient: _finish_hunting 0 2024-04-04T21:39:17.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.246+0000 7f1e2ffff640 1 monclient: found mon.noname-c 2024-04-04T21:39:17.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.246+0000 7f1e2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:17.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.246+0000 7f1e2ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:39:17.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.246+0000 7f1e2ffff640 1 -- 172.21.15.106:0/2350258324 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e300aa250 con 0x7f1e300a9e30 2024-04-04T21:39:17.250 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.251+0000 7f1e2ffff640 1 --2- 172.21.15.106:0/2350258324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e300a9e30 0x7f1e30142880 secure :-1 s=READY pgs=5545 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c009870 tx=0x7f1e1c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaee5b65e1af609a server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:17.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e2f7fe640 1 -- 172.21.15.106:0/2350258324 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c035020 con 0x7f1e300a9e30 2024-04-04T21:39:17.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:17.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e2f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:39:17.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e2f7fe640 10 monclient: dump: 2024-04-04T21:39:17.255 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:17.255 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.255 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.255 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.255 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:17.256 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.256 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:17.256 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:17.256 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:17.256 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e2f7fe640 10 monclient: _finish_auth 0 2024-04-04T21:39:17.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e2f7fe640 10 monclient: _check_auth_tickets 2024-04-04T21:39:17.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:17.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e2f7fe640 1 -- 172.21.15.106:0/2350258324 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1c031870 con 0x7f1e300a9e30 2024-04-04T21:39:17.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:17.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e2f7fe640 1 -- 172.21.15.106:0/2350258324 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c031c40 con 0x7f1e300a9e30 2024-04-04T21:39:17.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:17.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e2f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:17.256 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e2f7fe640 10 monclient: dump: 2024-04-04T21:39:17.257 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:17.257 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.257 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.257 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.257 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:17.257 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.257 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:17.257 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:17.257 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:17.257 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e36a9e640 5 monclient: authenticate success, global_id 22090 2024-04-04T21:39:17.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e36a9e640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:17.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.255+0000 7f1e36a9e640 4 set_mon_vals no callback set 2024-04-04T21:39:17.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.256+0000 7f1e36a9e640 10 monclient: shutdown 2024-04-04T21:39:17.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.256+0000 7f1e36a9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:17.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.256+0000 7f1e36a9e640 1 -- 172.21.15.106:0/2350258324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e300a9e30 msgr2=0x7f1e30142880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.256+0000 7f1e36a9e640 1 --2- 172.21.15.106:0/2350258324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e300a9e30 0x7f1e30142880 secure :-1 s=READY pgs=5545 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c009870 tx=0x7f1e1c02b910 comp rx=0 tx=0).stop 2024-04-04T21:39:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.256+0000 7f1e36a9e640 1 -- 172.21.15.106:0/2350258324 shutdown_connections 2024-04-04T21:39:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.256+0000 7f1e36a9e640 1 --2- 172.21.15.106:0/2350258324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e300a9e30 0x7f1e30142880 unknown :-1 s=CLOSED pgs=5545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.256+0000 7f1e36a9e640 1 -- 172.21.15.106:0/2350258324 >> 172.21.15.106:0/2350258324 conn(0x7f1e300b8030 msgr2=0x7f1e300b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.256+0000 7f1e36a9e640 1 -- 172.21.15.106:0/2350258324 shutdown_connections 2024-04-04T21:39:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.256+0000 7f1e36a9e640 1 -- 172.21.15.106:0/2350258324 wait complete. 2024-04-04T21:39:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.256+0000 7f1e36a9e640 10 monclient: build_initial_monmap 2024-04-04T21:39:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.256+0000 7f1e36a9e640 10 monclient: monmap: 2024-04-04T21:39:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:17.262 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:17.262 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:17.262 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 1 librados: starting msgr at 2024-04-04T21:39:17.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 1 librados: starting objecter 2024-04-04T21:39:17.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 1 Processor -- start 2024-04-04T21:39:17.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 1 -- start start 2024-04-04T21:39:17.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 1 librados: setting wanted keys 2024-04-04T21:39:17.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 1 librados: calling monclient init 2024-04-04T21:39:17.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 10 monclient: init 2024-04-04T21:39:17.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:17.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-04T21:39:17.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e300a9a50 0x7f1e30134ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 10 monclient(hunting): picked mon.noname-b con 0x7f1e300a9a50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e300a9e30 0x7f1e30140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 10 monclient(hunting): picked mon.noname-a con 0x7f1e300a9e30 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30142dc0 0x7f1e30143ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 10 monclient(hunting): picked mon.noname-c con 0x7f1e30142dc0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e301433b0 con 0x7f1e300a9e30 2024-04-04T21:39:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e30143230 con 0x7f1e300a9a50 2024-04-04T21:39:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3014e610 con 0x7f1e30142dc0 2024-04-04T21:39:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.257+0000 7f1e36a9e640 10 monclient(hunting): authenticate will time out at 3606.525111s 2024-04-04T21:39:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.258+0000 7f1e3503b640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30142dc0 0x7f1e30143ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.258+0000 7f1e3483a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e300a9a50 0x7f1e30134ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3483a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e300a9a50 0x7f1e30134ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:50284/0 (socket says 172.21.15.106:50284) 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3483a640 1 -- 172.21.15.106:0/3465924497 learned_addr learned my addr 172.21.15.106:0/3465924497 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3483a640 10 monclient(hunting): get_auth_request con 0x7f1e300a9a50 auth_method 0 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3483a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3483a640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3483a640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3503b640 10 monclient(hunting): get_auth_request con 0x7f1e30142dc0 auth_method 0 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3503b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3503b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3503b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3483a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3483a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3483a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3503b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3503b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:17.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.259+0000 7f1e3503b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:17.265 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.262+0000 7f1e3483a640 10 monclient(hunting): handle_auth_done global_id 22093 payload 824 2024-04-04T21:39:17.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.262+0000 7f1e3483a640 1 -- 172.21.15.106:0/3465924497 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30142dc0 msgr2=0x7f1e30143ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.262+0000 7f1e3483a640 1 --2- 172.21.15.106:0/3465924497 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30142dc0 0x7f1e30143ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.262+0000 7f1e3483a640 1 -- 172.21.15.106:0/3465924497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e300a9e30 msgr2=0x7f1e30140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.262+0000 7f1e3483a640 1 --2- 172.21.15.106:0/3465924497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e300a9e30 0x7f1e30140eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.262+0000 7f1e3483a640 10 monclient: _finish_hunting 0 2024-04-04T21:39:17.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.262+0000 7f1e3483a640 1 monclient: found mon.noname-b 2024-04-04T21:39:17.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.262+0000 7f1e3483a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:17.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.262+0000 7f1e3483a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:17.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.262+0000 7f1e3483a640 1 -- 172.21.15.106:0/3465924497 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e30145a40 con 0x7f1e300a9a50 2024-04-04T21:39:17.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.264+0000 7f1e3483a640 1 --2- 172.21.15.106:0/3465924497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e300a9a50 0x7f1e30134ad0 secure :-1 s=READY pgs=5546 cs=0 l=1 rev1=1 crypto rx=0x7f1e2400c400 tx=0x7f1e2400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:17.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.264+0000 7f1e2d7fa640 10 client.?.objecter ms_handle_connect 0x7f1e300a9a50 2024-04-04T21:39:17.266 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.264+0000 7f1e2d7fa640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:17.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e2d7fa640 1 -- 172.21.15.106:0/3465924497 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e240052d0 con 0x7f1e300a9a50 2024-04-04T21:39:17.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:17.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e2d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:17.267 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e2d7fa640 10 monclient: dump: 2024-04-04T21:39:17.268 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:17.268 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.268 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.268 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.268 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:17.268 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.268 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:17.268 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:17.268 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:17.268 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.268 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e2d7fa640 10 monclient: _finish_auth 0 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e2d7fa640 10 monclient: _check_auth_tickets 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e2d7fa640 1 -- 172.21.15.106:0/3465924497 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e24005490 con 0x7f1e300a9a50 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e2d7fa640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e2d7fa640 1 -- 172.21.15.106:0/3465924497 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24007260 con 0x7f1e300a9a50 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e36a9e640 5 monclient: authenticate success, global_id 22093 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e36a9e640 10 monclient: _renew_subs 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e36a9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e36a9e640 1 -- 172.21.15.106:0/3465924497 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e30145d20 con 0x7f1e300a9a50 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e2effd640 4 set_mon_vals no callback set 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e2d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e2d7fa640 10 monclient: dump: 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:17.269 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.270 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.270 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.270 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:17.270 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.270 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:17.270 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:17.270 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:17.270 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e36a9e640 10 client.22093.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:17.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e36a9e640 10 monclient: _renew_subs 2024-04-04T21:39:17.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e36a9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:17.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e36a9e640 1 -- 172.21.15.106:0/3465924497 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e30144000 con 0x7f1e300a9a50 2024-04-04T21:39:17.270 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.267+0000 7f1e36a9e640 1 librados: init done 2024-04-04T21:39:17.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.268+0000 7f1e36a9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:17.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.268+0000 7f1e36a9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:17.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.268+0000 7f1e36a9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:17.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.268+0000 7f1e36a9e640 1 -- 172.21.15.106:0/3465924497 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3004f050 con 0x7f1e300a9a50 2024-04-04T21:39:17.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.285+0000 7f1e2d7fa640 1 -- 172.21.15.106:0/3465924497 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1e24007400 con 0x7f1e300a9a50 2024-04-04T21:39:17.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.285+0000 7f1e2d7fa640 1 --2- 172.21.15.106:0/3465924497 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f1e1406ce60 0x7f1e1406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.286+0000 7f1e2ffff640 1 --2- 172.21.15.106:0/3465924497 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f1e1406ce60 0x7f1e1406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.286+0000 7f1e2ffff640 10 monclient: get_auth_request con 0x7f1e1406ce60 auth_method 0 2024-04-04T21:39:17.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.286+0000 7f1e2d7fa640 10 client.22093.objecter ms_handle_connect 0x7f1e1406ce60 2024-04-04T21:39:17.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.286+0000 7f1e2ffff640 1 --2- 172.21.15.106:0/3465924497 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f1e1406ce60 0x7f1e1406f320 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c036000 tx=0x7f1e1c030000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:17.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.518+0000 7f1e2d7fa640 1 -- 172.21.15.106:0/3465924497 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f1e240564b0 con 0x7f1e300a9a50 2024-04-04T21:39:17.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.518+0000 7f1e2d7fa640 10 client.22093.objecter ms_dispatch 0x7f1e30000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:17.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.518+0000 7f1e2d7fa640 3 client.22093.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:17.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.518+0000 7f1e2d7fa640 3 client.22093.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:17.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.518+0000 7f1e2d7fa640 20 client.22093.objecter dump_active .. 0 homeless 2024-04-04T21:39:17.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.528+0000 7f1e2d7fa640 1 -- 172.21.15.106:0/3465924497 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1e24058410 con 0x7f1e300a9a50 2024-04-04T21:39:17.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.528+0000 7f1e2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:17.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.528+0000 7f1e2d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:17.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.704+0000 7f1e36a9e640 1 -- 172.21.15.106:0/3465924497 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e3013db90 con 0x7f1e1406ce60 2024-04-04T21:39:17.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.705+0000 7f1e2d7fa640 1 -- 172.21.15.106:0/3465924497 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1e3013db90 con 0x7f1e1406ce60 2024-04-04T21:39:17.706 INFO:teuthology.orchestra.run.smithi106.stderr:instructed osd(s) 3 to deep-scrub 2024-04-04T21:39:17.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.706+0000 7f1e36a9e640 10 librados: watch_flush enter 2024-04-04T21:39:17.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 10 librados: watch_flush exit 2024-04-04T21:39:17.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 20 client.22093.objecter shutdown clearing up homeless session... 2024-04-04T21:39:17.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 10 client.22093.objecter successfully canceled tick 2024-04-04T21:39:17.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 1 -- 172.21.15.106:0/3465924497 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f1e1406ce60 msgr2=0x7f1e1406f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 1 --2- 172.21.15.106:0/3465924497 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f1e1406ce60 0x7f1e1406f320 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c036000 tx=0x7f1e1c030000 comp rx=0 tx=0).stop 2024-04-04T21:39:17.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 10 monclient: shutdown 2024-04-04T21:39:17.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:17.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 1 -- 172.21.15.106:0/3465924497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e300a9a50 msgr2=0x7f1e30134ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 1 --2- 172.21.15.106:0/3465924497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e300a9a50 0x7f1e30134ad0 secure :-1 s=READY pgs=5546 cs=0 l=1 rev1=1 crypto rx=0x7f1e2400c400 tx=0x7f1e2400c8c0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 1 -- 172.21.15.106:0/3465924497 shutdown_connections 2024-04-04T21:39:17.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 1 --2- 172.21.15.106:0/3465924497 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f1e1406ce60 0x7f1e1406f320 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 1 --2- 172.21.15.106:0/3465924497 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30142dc0 0x7f1e30143ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 1 --2- 172.21.15.106:0/3465924497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e300a9e30 0x7f1e30140eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 1 --2- 172.21.15.106:0/3465924497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e300a9a50 0x7f1e30134ad0 unknown :-1 s=CLOSED pgs=5546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 1 -- 172.21.15.106:0/3465924497 >> 172.21.15.106:0/3465924497 conn(0x7f1e300b8030 msgr2=0x7f1e30138710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:17.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 1 -- 172.21.15.106:0/3465924497 shutdown_connections 2024-04-04T21:39:17.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 1 -- 172.21.15.106:0/3465924497 wait complete. 2024-04-04T21:39:17.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.707+0000 7f1e36a9e640 1 librados: shutdown 2024-04-04T21:39:17.720 INFO:tasks.ceph:Scrubbing osd.4 2024-04-04T21:39:17.721 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-04T21:39:17.824 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.824+0000 7fe1f3101640 10 monclient: get_monmap_and_config 2024-04-04T21:39:17.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.824+0000 7fe1f3101640 10 monclient: build_initial_monmap 2024-04-04T21:39:17.825 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.824+0000 7fe1f3101640 10 monclient: monmap: 2024-04-04T21:39:17.825 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:17.825 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.825 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:17.825607+0000 2024-04-04T21:39:17.825 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:17.825607+0000 2024-04-04T21:39:17.825 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:17.825 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.826 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:17.826 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:17.826 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:17.826 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:17.826 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:17.826 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 1 Processor -- start 2024-04-04T21:39:17.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 1 -- start start 2024-04-04T21:39:17.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 10 monclient: init 2024-04-04T21:39:17.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:17.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-04T21:39:17.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe1ec071050 0x7fe1ec0b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 10 monclient(hunting): picked mon.noname-a con 0x7fe1ec071050 addr v2:172.21.15.106:3300/0 2024-04-04T21:39:17.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe1ec140ae0 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:17.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe1ec140f20 addr v1:172.21.15.203:6789/0 2024-04-04T21:39:17.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1ec072fc0 con 0x7fe1ec140ae0 2024-04-04T21:39:17.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1ec13de50 con 0x7fe1ec140f20 2024-04-04T21:39:17.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:17.829 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1ec071750 con 0x7fe1ec071050 2024-04-04T21:39:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.825+0000 7fe1f3101640 10 monclient(hunting): authenticate will time out at 3607.093162s 2024-04-04T21:39:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.826+0000 7fe1f0e9d640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe1ec071050 0x7fe1ec0b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.826+0000 7fe1f169e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fe1ec140f20 0x7fe1ec14ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:55698/0 (socket says 172.21.15.106:55698) 2024-04-04T21:39:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.826+0000 7fe1f169e640 1 -- 172.21.15.106:0/368248239 learned_addr learned my addr 172.21.15.106:0/368248239 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:39:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.826+0000 7fe1f0e9d640 10 monclient(hunting): get_auth_request con 0x7fe1ec071050 auth_method 0 2024-04-04T21:39:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.826+0000 7fe1f0e9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.826+0000 7fe1f0e9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.826+0000 7fe1f0e9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.827+0000 7fe1f0e9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.827+0000 7fe1f0e9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.827+0000 7fe1f0e9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.829+0000 7fe1f0e9d640 10 monclient(hunting): handle_auth_done global_id 15138 payload 274 2024-04-04T21:39:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.829+0000 7fe1f0e9d640 1 -- 172.21.15.106:0/368248239 >> v1:172.21.15.203:6789/0 conn(0x7fe1ec140f20 legacy=0x7fe1ec14ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.829+0000 7fe1f0e9d640 1 -- 172.21.15.106:0/368248239 >> v1:172.21.15.106:6789/0 conn(0x7fe1ec140ae0 legacy=0x7fe1ec13d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:17.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.829+0000 7fe1f0e9d640 10 monclient: _finish_hunting 0 2024-04-04T21:39:17.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.829+0000 7fe1f0e9d640 1 monclient: found mon.noname-a 2024-04-04T21:39:17.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.829+0000 7fe1f0e9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:17.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.829+0000 7fe1f0e9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:39:17.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.829+0000 7fe1f0e9d640 1 -- 172.21.15.106:0/368248239 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec0723d0 con 0x7fe1ec071050 2024-04-04T21:39:17.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.830+0000 7fe1f0e9d640 1 --2- 172.21.15.106:0/368248239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1ec071050 0x7fe1ec0b0a10 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00097a0 tx=0x7fe1e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fdfafc58d9c381e server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:17.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.836+0000 7fe1eb7fe640 1 -- 172.21.15.106:0/368248239 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0035020 con 0x7fe1ec071050 2024-04-04T21:39:17.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.836+0000 7fe1eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:17.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:39:17.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1eb7fe640 10 monclient: dump: 2024-04-04T21:39:17.837 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:17.837 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.837 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.837 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.838 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:17.838 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.838 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:17.838 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:17.838 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:17.838 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1eb7fe640 10 monclient: _finish_auth 0 2024-04-04T21:39:17.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1eb7fe640 10 monclient: _check_auth_tickets 2024-04-04T21:39:17.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:17.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1eb7fe640 1 -- 172.21.15.106:0/368248239 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e0031830 con 0x7fe1ec071050 2024-04-04T21:39:17.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1f3101640 5 monclient: authenticate success, global_id 15138 2024-04-04T21:39:17.839 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1f3101640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:39:17.839 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:17.839 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1f3101640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:17.839 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1eb7fe640 1 -- 172.21.15.106:0/368248239 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0031c00 con 0x7fe1ec071050 2024-04-04T21:39:17.839 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:17.839 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1f3101640 4 set_mon_vals no callback set 2024-04-04T21:39:17.839 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:39:17.839 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1eb7fe640 10 monclient: dump: 2024-04-04T21:39:17.839 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:17.839 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.839 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.839 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.840 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:17.840 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.840 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:17.840 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:17.840 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:17.840 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1f3101640 10 monclient: shutdown 2024-04-04T21:39:17.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1f3101640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:17.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1f3101640 1 -- 172.21.15.106:0/368248239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1ec071050 msgr2=0x7fe1ec0b0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1f3101640 1 --2- 172.21.15.106:0/368248239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1ec071050 0x7fe1ec0b0a10 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00097a0 tx=0x7fe1e002b8d0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.844 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1f3101640 1 -- 172.21.15.106:0/368248239 shutdown_connections 2024-04-04T21:39:17.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1f3101640 1 --2- 172.21.15.106:0/368248239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1ec071050 0x7fe1ec0b0a10 unknown :-1 s=CLOSED pgs=3164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1f3101640 1 -- 172.21.15.106:0/368248239 >> 172.21.15.106:0/368248239 conn(0x7fe1ec0b6160 msgr2=0x7fe1ec0b6570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:17.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1f3101640 1 -- 172.21.15.106:0/368248239 shutdown_connections 2024-04-04T21:39:17.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.837+0000 7fe1f3101640 1 -- 172.21.15.106:0/368248239 wait complete. 2024-04-04T21:39:17.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.838+0000 7fe1f3101640 10 monclient: build_initial_monmap 2024-04-04T21:39:17.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.838+0000 7fe1f3101640 10 monclient: monmap: 2024-04-04T21:39:17.845 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:17.845 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:17.845 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:17.845 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:17.845 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:17.845 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.845 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:17.845 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:17.846 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:17.846 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 1 librados: starting msgr at 2024-04-04T21:39:17.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 1 librados: starting objecter 2024-04-04T21:39:17.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 1 Processor -- start 2024-04-04T21:39:17.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 1 -- start start 2024-04-04T21:39:17.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 1 librados: setting wanted keys 2024-04-04T21:39:17.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 1 librados: calling monclient init 2024-04-04T21:39:17.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 10 monclient: init 2024-04-04T21:39:17.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:17.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:39:17.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe1ec071050 0x7fe1ec143960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 10 monclient(hunting): picked mon.noname-c con 0x7fe1ec071050 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:17.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1ec140ae0 0x7fe1ec1d7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 10 monclient(hunting): picked mon.noname-b con 0x7fe1ec140ae0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:17.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1ec140f20 0x7fe1ec1db990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 10 monclient(hunting): picked mon.noname-a con 0x7fe1ec140f20 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:17.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:17.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec13d940 con 0x7fe1ec140f20 2024-04-04T21:39:17.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:17.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec0732a0 con 0x7fe1ec140ae0 2024-04-04T21:39:17.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:17.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec13dac0 con 0x7fe1ec071050 2024-04-04T21:39:17.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:17.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.839+0000 7fe1f3101640 10 monclient(hunting): authenticate will time out at 3607.107091s 2024-04-04T21:39:17.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1f169e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1ec140f20 0x7fe1ec1db990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1f0e9d640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe1ec071050 0x7fe1ec143960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1ebfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1ec140ae0 0x7fe1ec1d7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1f169e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1ec140f20 0x7fe1ec1db990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42482/0 (socket says 172.21.15.106:42482) 2024-04-04T21:39:17.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1f169e640 1 -- 172.21.15.106:0/975987106 learned_addr learned my addr 172.21.15.106:0/975987106 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:17.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1f169e640 10 monclient(hunting): get_auth_request con 0x7fe1ec140f20 auth_method 0 2024-04-04T21:39:17.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1f169e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:17.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1f169e640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:17.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1f169e640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:17.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1f0e9d640 10 monclient(hunting): get_auth_request con 0x7fe1ec071050 auth_method 0 2024-04-04T21:39:17.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1f0e9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:17.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1f0e9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:17.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1f0e9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:17.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1ebfff640 10 monclient(hunting): get_auth_request con 0x7fe1ec140ae0 auth_method 0 2024-04-04T21:39:17.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:17.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:17.851 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.840+0000 7fe1ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:17.851 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.841+0000 7fe1f169e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:17.851 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.841+0000 7fe1f169e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:17.851 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.841+0000 7fe1f169e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:17.851 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.841+0000 7fe1ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:17.851 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.841+0000 7fe1ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:17.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.841+0000 7fe1ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:17.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.841+0000 7fe1f0e9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:17.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.841+0000 7fe1f0e9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:17.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.841+0000 7fe1f0e9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:17.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.844+0000 7fe1f169e640 10 monclient(hunting): handle_auth_done global_id 15141 payload 824 2024-04-04T21:39:17.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.844+0000 7fe1f169e640 1 -- 172.21.15.106:0/975987106 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe1ec071050 msgr2=0x7fe1ec143960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.844+0000 7fe1f169e640 1 --2- 172.21.15.106:0/975987106 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe1ec071050 0x7fe1ec143960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.844+0000 7fe1f169e640 1 -- 172.21.15.106:0/975987106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1ec140ae0 msgr2=0x7fe1ec1d7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.844+0000 7fe1f169e640 1 --2- 172.21.15.106:0/975987106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1ec140ae0 0x7fe1ec1d7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.844+0000 7fe1f169e640 10 monclient: _finish_hunting 0 2024-04-04T21:39:17.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.844+0000 7fe1f169e640 1 monclient: found mon.noname-a 2024-04-04T21:39:17.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.844+0000 7fe1f169e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:17.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.844+0000 7fe1f169e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:39:17.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.844+0000 7fe1f169e640 1 -- 172.21.15.106:0/975987106 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec06dc70 con 0x7fe1ec140f20 2024-04-04T21:39:17.854 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.844+0000 7fe1f0e9d640 1 --2- 172.21.15.106:0/975987106 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe1ec071050 0x7fe1ec143960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-04T21:39:17.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.846+0000 7fe1f169e640 1 --2- 172.21.15.106:0/975987106 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1ec140f20 0x7fe1ec1db990 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400cb10 tx=0x7fe1e400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:17.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.846+0000 7fe1e97fa640 10 client.?.objecter ms_handle_connect 0x7fe1ec140f20 2024-04-04T21:39:17.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.846+0000 7fe1e97fa640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:17.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1e97fa640 1 -- 172.21.15.106:0/975987106 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4017070 con 0x7fe1ec140f20 2024-04-04T21:39:17.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:17.857 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1e97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:39:17.858 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1e97fa640 10 monclient: dump: 2024-04-04T21:39:17.858 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:17.858 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.858 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.858 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.858 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:17.859 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.859 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:17.859 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:17.859 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:17.859 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.859 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1e97fa640 10 monclient: _finish_auth 0 2024-04-04T21:39:17.860 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1e97fa640 10 monclient: _check_auth_tickets 2024-04-04T21:39:17.860 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:17.860 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1e97fa640 1 -- 172.21.15.106:0/975987106 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e400ac70 con 0x7fe1ec140f20 2024-04-04T21:39:17.860 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1e97fa640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:17.860 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1f3101640 5 monclient: authenticate success, global_id 15141 2024-04-04T21:39:17.860 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1e97fa640 1 -- 172.21.15.106:0/975987106 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4012600 con 0x7fe1ec140f20 2024-04-04T21:39:17.860 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1f3101640 10 monclient: _renew_subs 2024-04-04T21:39:17.861 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1f3101640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:39:17.861 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1f3101640 1 -- 172.21.15.106:0/975987106 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1ec0af550 con 0x7fe1ec140f20 2024-04-04T21:39:17.861 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1eaffd640 4 set_mon_vals no callback set 2024-04-04T21:39:17.861 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:17.861 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1e97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:39:17.861 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1e97fa640 10 monclient: dump: 2024-04-04T21:39:17.862 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:17.862 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:17.862 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.862 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:17.862 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:17.862 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:17.863 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:17.863 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:17.863 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:17.863 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:17.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1f3101640 10 client.15141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:17.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1f3101640 10 monclient: _renew_subs 2024-04-04T21:39:17.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1f3101640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:39:17.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1f3101640 1 -- 172.21.15.106:0/975987106 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1ec1dc0b0 con 0x7fe1ec140f20 2024-04-04T21:39:17.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.849+0000 7fe1f3101640 1 librados: init done 2024-04-04T21:39:17.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.850+0000 7fe1f3101640 10 client.15141.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:17.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.850+0000 7fe1f3101640 15 client.15141.objecter _session_command_op_assign -1 1 2024-04-04T21:39:17.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.850+0000 7fe1f3101640 20 client.15141.objecter _get_session osd=-1 returning homeless 2024-04-04T21:39:17.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.850+0000 7fe1f3101640 10 client.15141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:17.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.850+0000 7fe1f3101640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:39:17.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.850+0000 7fe1f3101640 1 -- 172.21.15.106:0/975987106 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe1ec0685a0 con 0x7fe1ec140f20 2024-04-04T21:39:17.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.869+0000 7fe1e97fa640 1 -- 172.21.15.106:0/975987106 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe1e401a070 con 0x7fe1ec140f20 2024-04-04T21:39:17.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.869+0000 7fe1e97fa640 1 --2- 172.21.15.106:0/975987106 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe1d006ceb0 0x7fe1d006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.869+0000 7fe1f0e9d640 1 --2- 172.21.15.106:0/975987106 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe1d006ceb0 0x7fe1d006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.869+0000 7fe1f0e9d640 10 monclient: get_auth_request con 0x7fe1d006ceb0 auth_method 0 2024-04-04T21:39:17.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.870+0000 7fe1e97fa640 10 client.15141.objecter ms_handle_connect 0x7fe1d006ceb0 2024-04-04T21:39:17.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.870+0000 7fe1f0e9d640 1 --2- 172.21.15.106:0/975987106 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe1d006ceb0 0x7fe1d006f370 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0036000 tx=0x7fe1e003c040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:17.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 1 -- 172.21.15.106:0/975987106 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7fe1e4056e60 con 0x7fe1ec140f20 2024-04-04T21:39:17.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 10 client.15141.objecter ms_dispatch 0x7fe1ec000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:17.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 3 client.15141.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:17.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 3 client.15141.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:17.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 10 client.15141.objecter checking command 1 2024-04-04T21:39:17.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 1 --2- 172.21.15.106:0/975987106 >> [v2:172.21.15.203:6824/2225997929,v1:172.21.15.203:6825/2225997929] conn(0x7fe1d0074bc0 0x7fe1d0077060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:17.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 20 client.15141.objecter _get_session s=0x7fe1d00744f0 osd=4 3 2024-04-04T21:39:17.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 20 client.15141.objecter put_session s=0x7fe1d00744f0 osd=4 3 2024-04-04T21:39:17.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 15 client.15141.objecter _session_command_op_remove -1 1 2024-04-04T21:39:17.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 20 client.15141.objecter _get_session s=0x7fe1d00744f0 osd=4 3 2024-04-04T21:39:17.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 20 client.15141.objecter get_session s=0x7fe1d00744f0 osd=4 3 2024-04-04T21:39:17.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 15 client.15141.objecter _session_command_op_assign 4 1 2024-04-04T21:39:17.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 20 client.15141.objecter put_session s=0x7fe1d00744f0 osd=4 4 2024-04-04T21:39:17.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 10 client.15141.objecter _send_command 1 2024-04-04T21:39:17.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 1 -- 172.21.15.106:0/975987106 --> [v2:172.21.15.203:6824/2225997929,v1:172.21.15.203:6825/2225997929] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1e401de80 con 0x7fe1d0074bc0 2024-04-04T21:39:17.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 20 client.15141.objecter dump_active .. 0 homeless 2024-04-04T21:39:17.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 1 -- 172.21.15.106:0/975987106 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=1197) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1e4055060 con 0x7fe1ec140f20 2024-04-04T21:39:17.873 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.872+0000 7fe1e97fa640 10 monclient: handle_get_version_reply finishing 1 version 1197 2024-04-04T21:39:17.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.873+0000 7fe1ebfff640 1 --2- 172.21.15.106:0/975987106 >> [v2:172.21.15.203:6824/2225997929,v1:172.21.15.203:6825/2225997929] conn(0x7fe1d0074bc0 0x7fe1d0077060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:17.874 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.873+0000 7fe1ebfff640 10 monclient: get_auth_request con 0x7fe1d0074bc0 auth_method 0 2024-04-04T21:39:17.875 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.876+0000 7fe1ebfff640 1 --2- 172.21.15.106:0/975987106 >> [v2:172.21.15.203:6824/2225997929,v1:172.21.15.203:6825/2225997929] conn(0x7fe1d0074bc0 0x7fe1d0077060 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:17.876 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.876+0000 7fe1e97fa640 10 client.15141.objecter ms_handle_connect 0x7fe1d0074bc0 2024-04-04T21:39:17.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.899+0000 7fe1e97fa640 1 -- 172.21.15.106:0/975987106 <== osd.4 v2:172.21.15.203:6824/2225997929 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe1e401de80 con 0x7fe1d0074bc0 2024-04-04T21:39:17.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.899+0000 7fe1e97fa640 10 client.15141.objecter ms_dispatch 0x7fe1ec000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:39:17.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.899+0000 7fe1e97fa640 10 client.15141.objecter _finish_command 1 = system:0 2024-04-04T21:39:17.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.899+0000 7fe1e97fa640 20 client.15141.objecter put_session s=0x7fe1d00744f0 osd=4 4 2024-04-04T21:39:17.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.899+0000 7fe1e97fa640 15 client.15141.objecter _session_command_op_remove 4 1 2024-04-04T21:39:17.912 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.912+0000 7fe1f3101640 10 client.15141.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-04T21:39:17.912 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.912+0000 7fe1f3101640 15 client.15141.objecter _session_command_op_assign -1 2 2024-04-04T21:39:17.912 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.912+0000 7fe1f3101640 20 client.15141.objecter _get_session s=0x7fe1d00744f0 osd=4 3 2024-04-04T21:39:17.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.912+0000 7fe1f3101640 20 client.15141.objecter put_session s=0x7fe1d00744f0 osd=4 3 2024-04-04T21:39:17.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.912+0000 7fe1f3101640 20 client.15141.objecter _get_session s=0x7fe1d00744f0 osd=4 3 2024-04-04T21:39:17.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.912+0000 7fe1f3101640 15 client.15141.objecter _session_command_op_remove -1 2 2024-04-04T21:39:17.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.912+0000 7fe1f3101640 20 client.15141.objecter get_session s=0x7fe1d00744f0 osd=4 3 2024-04-04T21:39:17.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.912+0000 7fe1f3101640 15 client.15141.objecter _session_command_op_assign 4 2 2024-04-04T21:39:17.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.912+0000 7fe1f3101640 20 client.15141.objecter put_session s=0x7fe1d00744f0 osd=4 4 2024-04-04T21:39:17.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.912+0000 7fe1f3101640 10 client.15141.objecter _send_command 2 2024-04-04T21:39:17.913 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.912+0000 7fe1f3101640 1 -- 172.21.15.106:0/975987106 --> [v2:172.21.15.203:6824/2225997929,v1:172.21.15.203:6825/2225997929] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe1ec0af8d0 con 0x7fe1d0074bc0 2024-04-04T21:39:17.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1e97fa640 1 -- 172.21.15.106:0/975987106 <== osd.4 v2:172.21.15.203:6824/2225997929 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe1ec0af8d0 con 0x7fe1d0074bc0 2024-04-04T21:39:17.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1e97fa640 10 client.15141.objecter ms_dispatch 0x7fe1ec000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:39:17.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1e97fa640 10 client.15141.objecter _finish_command 2 = system:0 2024-04-04T21:39:17.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1e97fa640 20 client.15141.objecter put_session s=0x7fe1d00744f0 osd=4 4 2024-04-04T21:39:17.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1e97fa640 15 client.15141.objecter _session_command_op_remove 4 2 2024-04-04T21:39:17.927 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-04T21:39:17.927 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-04T21:39:17.927 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-04T21:39:17.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f3101640 10 librados: watch_flush enter 2024-04-04T21:39:17.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f3101640 10 librados: watch_flush exit 2024-04-04T21:39:17.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f3101640 10 client.15141.objecter close_session for osd.4 2024-04-04T21:39:17.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f3101640 1 -- 172.21.15.106:0/975987106 >> [v2:172.21.15.203:6824/2225997929,v1:172.21.15.203:6825/2225997929] conn(0x7fe1d0074bc0 msgr2=0x7fe1d0077060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f3101640 1 --2- 172.21.15.106:0/975987106 >> [v2:172.21.15.203:6824/2225997929,v1:172.21.15.203:6825/2225997929] conn(0x7fe1d0074bc0 0x7fe1d0077060 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:17.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f3101640 20 client.15141.objecter put_session s=0x7fe1d00744f0 osd=4 1 2024-04-04T21:39:17.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f3101640 20 client.15141.objecter shutdown clearing up homeless session... 2024-04-04T21:39:17.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f3101640 10 client.15141.objecter successfully canceled tick 2024-04-04T21:39:17.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f3101640 1 -- 172.21.15.106:0/975987106 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe1d006ceb0 msgr2=0x7fe1d006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f3101640 1 --2- 172.21.15.106:0/975987106 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fe1d006ceb0 0x7fe1d006f370 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0036000 tx=0x7fe1e003c040 comp rx=0 tx=0).stop 2024-04-04T21:39:17.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f3101640 10 monclient: shutdown 2024-04-04T21:39:17.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f3101640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:17.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f3101640 1 -- 172.21.15.106:0/975987106 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1ec140f20 msgr2=0x7fe1ec1db990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:17.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f3101640 1 --2- 172.21.15.106:0/975987106 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1ec140f20 0x7fe1ec1db990 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400cb10 tx=0x7fe1e400a060 comp rx=0 tx=0).stop 2024-04-04T21:39:17.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.925+0000 7fe1f169e640 1 -- 172.21.15.106:0/975987106 reap_dead start 2024-04-04T21:39:17.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.926+0000 7fe1f3101640 1 -- 172.21.15.106:0/975987106 shutdown_connections 2024-04-04T21:39:17.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.926+0000 7fe1f3101640 1 -- 172.21.15.106:0/975987106 >> 172.21.15.106:0/975987106 conn(0x7fe1ec0b6160 msgr2=0x7fe1ec13d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:17.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.926+0000 7fe1f3101640 1 -- 172.21.15.106:0/975987106 shutdown_connections 2024-04-04T21:39:17.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.926+0000 7fe1f3101640 1 -- 172.21.15.106:0/975987106 wait complete. 2024-04-04T21:39:17.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:17.926+0000 7fe1f3101640 1 librados: shutdown 2024-04-04T21:39:17.938 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-04T21:39:18.038 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.038+0000 7f4ac3a40640 10 monclient: get_monmap_and_config 2024-04-04T21:39:18.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.038+0000 7f4ac3a40640 10 monclient: build_initial_monmap 2024-04-04T21:39:18.039 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.038+0000 7f4ac3a40640 10 monclient: monmap: 2024-04-04T21:39:18.039 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:18.039 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.039 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:18.039600+0000 2024-04-04T21:39:18.039 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:18.039600+0000 2024-04-04T21:39:18.039 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:18.039 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.039 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:18.039 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:18.039 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:18.039 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:18.039 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:18.040 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.040 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 1 Processor -- start 2024-04-04T21:39:18.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 1 -- start start 2024-04-04T21:39:18.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 10 monclient: init 2024-04-04T21:39:18.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:18.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-04-04T21:39:18.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4abc0b5200 addr v1:172.21.15.203:6789/0 2024-04-04T21:39:18.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4abc0a9aa0 0x7f4abc145af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 10 monclient(hunting): picked mon.noname-b con 0x7f4abc0a9aa0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:18.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4abc146030 0x7f4abc14ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 10 monclient(hunting): picked mon.noname-c con 0x7f4abc146030 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:18.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4abc0a97e0 con 0x7f4abc0b5200 2024-04-04T21:39:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4abc0b55e0 con 0x7f4abc146030 2024-04-04T21:39:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc0aa070 con 0x7f4abc0a9aa0 2024-04-04T21:39:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.039+0000 7f4ac3a40640 10 monclient(hunting): authenticate will time out at 3607.307069s 2024-04-04T21:39:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.040+0000 7f4ac0fdb640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4abc0a9aa0 0x7f4abc145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.040+0000 7f4ac1fdd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4abc146030 0x7f4abc14ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.040+0000 7f4ac0fdb640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4abc0a9aa0 0x7f4abc145af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:38188/0 (socket says 172.21.15.106:38188) 2024-04-04T21:39:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.040+0000 7f4ac0fdb640 1 -- 172.21.15.106:0/1343553767 learned_addr learned my addr 172.21.15.106:0/1343553767 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.040+0000 7f4ac0fdb640 10 monclient(hunting): get_auth_request con 0x7f4abc0a9aa0 auth_method 0 2024-04-04T21:39:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.040+0000 7f4ac0fdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.040+0000 7f4ac0fdb640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.040+0000 7f4ac0fdb640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.041+0000 7f4ac0fdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:18.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.041+0000 7f4ac0fdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:18.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.041+0000 7f4ac0fdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:18.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.041+0000 7f4ac1fdd640 10 monclient(hunting): get_auth_request con 0x7f4abc146030 auth_method 0 2024-04-04T21:39:18.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.041+0000 7f4ac1fdd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:18.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.041+0000 7f4ac1fdd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.041+0000 7f4ac1fdd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.042+0000 7f4ac1fdd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:18.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.042+0000 7f4ac1fdd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:18.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.042+0000 7f4ac1fdd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:18.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.042+0000 7f4ac0fdb640 10 monclient(hunting): handle_auth_done global_id 17678 payload 274 2024-04-04T21:39:18.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.042+0000 7f4ac0fdb640 1 -- 172.21.15.106:0/1343553767 >> v2:172.21.15.203:3300/0 conn(0x7f4abc146030 msgr2=0x7f4abc14ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:18.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.042+0000 7f4ac0fdb640 1 --2- 172.21.15.106:0/1343553767 >> v2:172.21.15.203:3300/0 conn(0x7f4abc146030 0x7f4abc14ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.043+0000 7f4ac0fdb640 1 -- 172.21.15.106:0/1343553767 >> v1:172.21.15.203:6789/0 conn(0x7f4abc0b5200 legacy=0x7f4abc13f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.043+0000 7f4ac0fdb640 10 monclient: _finish_hunting 0 2024-04-04T21:39:18.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.043+0000 7f4ac0fdb640 1 monclient: found mon.noname-b 2024-04-04T21:39:18.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.043+0000 7f4ac0fdb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:18.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.043+0000 7f4ac0fdb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:39:18.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.043+0000 7f4ac0fdb640 1 -- 172.21.15.106:0/1343553767 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc0ab420 con 0x7f4abc0a9aa0 2024-04-04T21:39:18.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.044+0000 7f4ac0fdb640 1 --2- 172.21.15.106:0/1343553767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4abc0a9aa0 0x7f4abc145af0 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7f4aac009870 tx=0x7f4aac02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=421d92a8be99e89f server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:18.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.046+0000 7f4ab3fff640 1 -- 172.21.15.106:0/1343553767 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac035020 con 0x7f4abc0a9aa0 2024-04-04T21:39:18.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.046+0000 7f4ab3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:18.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.046+0000 7f4ab3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:18.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.046+0000 7f4ab3fff640 10 monclient: dump: 2024-04-04T21:39:18.047 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:18.047 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.047 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.047 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.047 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:18.047 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.047 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:18.047 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:18.047 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:18.047 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.046+0000 7f4ab3fff640 10 monclient: _finish_auth 0 2024-04-04T21:39:18.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.046+0000 7f4ab3fff640 10 monclient: _check_auth_tickets 2024-04-04T21:39:18.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.046+0000 7f4ab3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:18.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.046+0000 7f4ac3a40640 5 monclient: authenticate success, global_id 17678 2024-04-04T21:39:18.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.046+0000 7f4ac3a40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:39:18.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ab3fff640 1 -- 172.21.15.106:0/1343553767 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aac031640 con 0x7f4abc0a9aa0 2024-04-04T21:39:18.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ab3fff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:18.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ab3fff640 1 -- 172.21.15.106:0/1343553767 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac02d040 con 0x7f4abc0a9aa0 2024-04-04T21:39:18.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ac3a40640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:18.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ac3a40640 4 set_mon_vals no callback set 2024-04-04T21:39:18.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ab3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:18.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ab3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ab3fff640 10 monclient: dump: 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ac3a40640 10 monclient: shutdown 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ac3a40640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1343553767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4abc0a9aa0 msgr2=0x7f4abc145af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ac3a40640 1 --2- 172.21.15.106:0/1343553767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4abc0a9aa0 0x7f4abc145af0 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7f4aac009870 tx=0x7f4aac02b8f0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1343553767 shutdown_connections 2024-04-04T21:39:18.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ac3a40640 1 --2- 172.21.15.106:0/1343553767 >> v2:172.21.15.203:3300/0 conn(0x7f4abc146030 0x7f4abc14ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ac3a40640 1 --2- 172.21.15.106:0/1343553767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4abc0a9aa0 0x7f4abc145af0 unknown :-1 s=CLOSED pgs=3270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1343553767 >> 172.21.15.106:0/1343553767 conn(0x7f4abc066e40 msgr2=0x7f4abc067250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:18.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.048+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1343553767 shutdown_connections 2024-04-04T21:39:18.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.049+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1343553767 wait complete. 2024-04-04T21:39:18.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.049+0000 7f4ac3a40640 10 monclient: build_initial_monmap 2024-04-04T21:39:18.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.049+0000 7f4ac3a40640 10 monclient: monmap: 2024-04-04T21:39:18.054 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:18.054 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:18.055 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:18.055 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:18.055 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:18.055 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.055 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:18.055 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:18.055 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:18.055 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.049+0000 7f4ac3a40640 1 librados: starting msgr at 2024-04-04T21:39:18.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.049+0000 7f4ac3a40640 1 librados: starting objecter 2024-04-04T21:39:18.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.049+0000 7f4ac3a40640 1 Processor -- start 2024-04-04T21:39:18.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 1 -- start start 2024-04-04T21:39:18.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 1 librados: setting wanted keys 2024-04-04T21:39:18.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 1 librados: calling monclient init 2024-04-04T21:39:18.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 10 monclient: init 2024-04-04T21:39:18.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:18.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:39:18.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4abc0a9aa0 0x7f4abc144000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 10 monclient(hunting): picked mon.noname-c con 0x7f4abc0a9aa0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:18.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4abc0b5200 0x7f4abc1d84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 10 monclient(hunting): picked mon.noname-b con 0x7f4abc0b5200 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:18.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4abc146030 0x7f4abc141e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 10 monclient(hunting): picked mon.noname-a con 0x7f4abc146030 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:18.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:18.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc1400c0 con 0x7f4abc146030 2024-04-04T21:39:18.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:18.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc0ab7c0 con 0x7f4abc0b5200 2024-04-04T21:39:18.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:18.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc140240 con 0x7f4abc0a9aa0 2024-04-04T21:39:18.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:18.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.050+0000 7f4ac3a40640 10 monclient(hunting): authenticate will time out at 3607.317719s 2024-04-04T21:39:18.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac17dc640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4abc0a9aa0 0x7f4abc144000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac0fdb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4abc0b5200 0x7f4abc1d84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac1fdd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4abc146030 0x7f4abc141e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac17dc640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4abc0a9aa0 0x7f4abc144000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:38192/0 (socket says 172.21.15.106:38192) 2024-04-04T21:39:18.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac17dc640 1 -- 172.21.15.106:0/1489857184 learned_addr learned my addr 172.21.15.106:0/1489857184 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:18.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac17dc640 10 monclient(hunting): get_auth_request con 0x7f4abc0a9aa0 auth_method 0 2024-04-04T21:39:18.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac17dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:18.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac17dc640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac17dc640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac0fdb640 10 monclient(hunting): get_auth_request con 0x7f4abc0b5200 auth_method 0 2024-04-04T21:39:18.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac0fdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:18.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac0fdb640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac0fdb640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac1fdd640 10 monclient(hunting): get_auth_request con 0x7f4abc146030 auth_method 0 2024-04-04T21:39:18.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac1fdd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:18.058 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac1fdd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac1fdd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac0fdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:18.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac0fdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:18.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac0fdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:18.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac17dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:18.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac17dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:18.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.051+0000 7f4ac17dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:18.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.052+0000 7f4ac1fdd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:18.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.052+0000 7f4ac1fdd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:18.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.052+0000 7f4ac1fdd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:18.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.054+0000 7f4ac0fdb640 10 monclient(hunting): handle_auth_done global_id 22108 payload 824 2024-04-04T21:39:18.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.054+0000 7f4ac0fdb640 1 -- 172.21.15.106:0/1489857184 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4abc0a9aa0 msgr2=0x7f4abc144000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.054+0000 7f4ac0fdb640 1 --2- 172.21.15.106:0/1489857184 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4abc0a9aa0 0x7f4abc144000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.054+0000 7f4ac0fdb640 1 -- 172.21.15.106:0/1489857184 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4abc146030 msgr2=0x7f4abc141e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.054+0000 7f4ac0fdb640 1 --2- 172.21.15.106:0/1489857184 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4abc146030 0x7f4abc141e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.054+0000 7f4ac0fdb640 10 monclient: _finish_hunting 0 2024-04-04T21:39:18.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.054+0000 7f4ac0fdb640 1 monclient: found mon.noname-b 2024-04-04T21:39:18.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.054+0000 7f4ac0fdb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:18.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.054+0000 7f4ac0fdb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:18.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.054+0000 7f4ac0fdb640 1 -- 172.21.15.106:0/1489857184 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc1d7f80 con 0x7f4abc0b5200 2024-04-04T21:39:18.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.056+0000 7f4ac0fdb640 1 --2- 172.21.15.106:0/1489857184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4abc0b5200 0x7f4abc1d84a0 secure :-1 s=READY pgs=5549 cs=0 l=1 rev1=1 crypto rx=0x7f4abc0aa1f0 tx=0x7f4aac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:18.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.056+0000 7f4ab1ffb640 10 client.?.objecter ms_handle_connect 0x7f4abc0b5200 2024-04-04T21:39:18.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.056+0000 7f4ab1ffb640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:18.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ab1ffb640 1 -- 172.21.15.106:0/1489857184 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac03aa40 con 0x7f4abc0b5200 2024-04-04T21:39:18.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ab1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:18.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ab1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:18.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ab1ffb640 10 monclient: dump: 2024-04-04T21:39:18.062 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:18.062 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:18.063 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ab1ffb640 10 monclient: _finish_auth 0 2024-04-04T21:39:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ab1ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:39:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ab1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ab1ffb640 1 -- 172.21.15.106:0/1489857184 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aac03ac00 con 0x7f4abc0b5200 2024-04-04T21:39:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ac3a40640 5 monclient: authenticate success, global_id 22108 2024-04-04T21:39:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ac3a40640 10 monclient: _renew_subs 2024-04-04T21:39:18.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ac3a40640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:18.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1489857184 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4abc1d7c30 con 0x7f4abc0b5200 2024-04-04T21:39:18.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ab1ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:18.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ab1ffb640 1 -- 172.21.15.106:0/1489857184 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac0443f0 con 0x7f4abc0b5200 2024-04-04T21:39:18.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ab1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:18.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ab1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:18.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ab1ffb640 10 monclient: dump: 2024-04-04T21:39:18.064 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:18.064 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.064 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.064 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.064 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:18.064 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.064 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:18.065 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:18.065 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:18.065 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ab37fe640 4 set_mon_vals no callback set 2024-04-04T21:39:18.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ac3a40640 10 client.22108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:18.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ac3a40640 10 monclient: _renew_subs 2024-04-04T21:39:18.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ac3a40640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:18.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1489857184 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4abc141120 con 0x7f4abc0b5200 2024-04-04T21:39:18.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.060+0000 7f4ac3a40640 1 librados: init done 2024-04-04T21:39:18.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.061+0000 7f4ac3a40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:18.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.061+0000 7f4ac3a40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:18.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.061+0000 7f4ac3a40640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:18.066 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.061+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1489857184 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4abc04f050 con 0x7f4abc0b5200 2024-04-04T21:39:18.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.310+0000 7f4ab1ffb640 1 -- 172.21.15.106:0/1489857184 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4aac038020 con 0x7f4abc0b5200 2024-04-04T21:39:18.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.311+0000 7f4ab1ffb640 1 --2- 172.21.15.106:0/1489857184 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f4aa006ceb0 0x7f4aa006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.311+0000 7f4ac17dc640 1 --2- 172.21.15.106:0/1489857184 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f4aa006ceb0 0x7f4aa006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.311+0000 7f4ac17dc640 10 monclient: get_auth_request con 0x7f4aa006ceb0 auth_method 0 2024-04-04T21:39:18.312 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.311+0000 7f4ac17dc640 1 --2- 172.21.15.106:0/1489857184 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f4aa006ceb0 0x7f4aa006f370 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8001840 tx=0x7f4ab8009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:18.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.311+0000 7f4ab1ffb640 10 client.22108.objecter ms_handle_connect 0x7f4aa006ceb0 2024-04-04T21:39:18.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.315+0000 7f4ab1ffb640 1 -- 172.21.15.106:0/1489857184 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f4aac078e50 con 0x7f4abc0b5200 2024-04-04T21:39:18.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.315+0000 7f4ab1ffb640 10 client.22108.objecter ms_dispatch 0x7f4abc000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:18.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.315+0000 7f4ab1ffb640 3 client.22108.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:18.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.315+0000 7f4ab1ffb640 3 client.22108.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:18.316 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.315+0000 7f4ab1ffb640 20 client.22108.objecter dump_active .. 0 homeless 2024-04-04T21:39:18.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.326+0000 7f4ab1ffb640 1 -- 172.21.15.106:0/1489857184 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4aac03f070 con 0x7f4abc0b5200 2024-04-04T21:39:18.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.326+0000 7f4ab1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:18.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.326+0000 7f4ab1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:18.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.516+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1489857184 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4abc1413b0 con 0x7f4aa006ceb0 2024-04-04T21:39:18.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.517+0000 7f4ab1ffb640 1 -- 172.21.15.106:0/1489857184 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4ab80014e0 con 0x7f4aa006ceb0 2024-04-04T21:39:18.517 INFO:teuthology.orchestra.run.smithi106.stderr:instructed osd(s) 4 to deep-scrub 2024-04-04T21:39:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.518+0000 7f4ac3a40640 10 librados: watch_flush enter 2024-04-04T21:39:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 10 librados: watch_flush exit 2024-04-04T21:39:18.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 20 client.22108.objecter shutdown clearing up homeless session... 2024-04-04T21:39:18.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 10 client.22108.objecter successfully canceled tick 2024-04-04T21:39:18.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1489857184 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f4aa006ceb0 msgr2=0x7f4aa006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 1 --2- 172.21.15.106:0/1489857184 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f4aa006ceb0 0x7f4aa006f370 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8001840 tx=0x7f4ab8009040 comp rx=0 tx=0).stop 2024-04-04T21:39:18.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 10 monclient: shutdown 2024-04-04T21:39:18.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:18.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1489857184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4abc0b5200 msgr2=0x7f4abc1d84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 1 --2- 172.21.15.106:0/1489857184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4abc0b5200 0x7f4abc1d84a0 secure :-1 s=READY pgs=5549 cs=0 l=1 rev1=1 crypto rx=0x7f4abc0aa1f0 tx=0x7f4aac0046c0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1489857184 shutdown_connections 2024-04-04T21:39:18.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 1 --2- 172.21.15.106:0/1489857184 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f4aa006ceb0 0x7f4aa006f370 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 1 --2- 172.21.15.106:0/1489857184 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4abc146030 0x7f4abc141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 1 --2- 172.21.15.106:0/1489857184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4abc0b5200 0x7f4abc1d84a0 unknown :-1 s=CLOSED pgs=5549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 1 --2- 172.21.15.106:0/1489857184 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4abc0a9aa0 0x7f4abc144000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1489857184 >> 172.21.15.106:0/1489857184 conn(0x7f4abc066e40 msgr2=0x7f4abc0b8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:18.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1489857184 shutdown_connections 2024-04-04T21:39:18.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 1 -- 172.21.15.106:0/1489857184 wait complete. 2024-04-04T21:39:18.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.519+0000 7f4ac3a40640 1 librados: shutdown 2024-04-04T21:39:18.530 INFO:tasks.ceph:Scrubbing osd.5 2024-04-04T21:39:18.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-04T21:39:18.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.633+0000 7f196bfb7640 10 monclient: get_monmap_and_config 2024-04-04T21:39:18.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.633+0000 7f196bfb7640 10 monclient: build_initial_monmap 2024-04-04T21:39:18.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.633+0000 7f196bfb7640 10 monclient: monmap: 2024-04-04T21:39:18.634 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:18.634 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.634 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:18.634739+0000 2024-04-04T21:39:18.634 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:18.634739+0000 2024-04-04T21:39:18.634 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:18.634 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.634 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:18.634 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:18.634 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:18.635 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:18.635 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:18.635 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.634+0000 7f196bfb7640 1 Processor -- start 2024-04-04T21:39:18.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.634+0000 7f196bfb7640 1 -- start start 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.634+0000 7f196bfb7640 10 monclient: init 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.634+0000 7f196bfb7640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.634+0000 7f196bfb7640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.634+0000 7f196bfb7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19640b5360 addr v1:172.21.15.203:6789/0 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.634+0000 7f196bfb7640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f196413fc70 0x7f1964145b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.634+0000 7f196bfb7640 10 monclient(hunting): picked mon.noname-b con 0x7f196413fc70 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.634+0000 7f196bfb7640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1964146090 0x7f196414cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.634+0000 7f196bfb7640 10 monclient(hunting): picked mon.noname-a con 0x7f1964146090 addr v2:172.21.15.106:3300/0 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.635+0000 7f196bfb7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19640ab3e0 con 0x7f19640b5360 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.635+0000 7f196bfb7640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.635+0000 7f196bfb7640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19640b5740 con 0x7f1964146090 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.635+0000 7f196bfb7640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.635+0000 7f196bfb7640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19640a9e70 con 0x7f196413fc70 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.635+0000 7f196bfb7640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:18.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.635+0000 7f196bfb7640 10 monclient(hunting): authenticate will time out at 3607.902209s 2024-04-04T21:39:18.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.635+0000 7f1969552640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f196413fc70 0x7f1964145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.635+0000 7f196a554640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1964146090 0x7f196414cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.635+0000 7f1969d53640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f19640b5360 0x7f196413f560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:55710/0 (socket says 172.21.15.106:55710) 2024-04-04T21:39:18.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.635+0000 7f1969d53640 1 -- 172.21.15.106:0/3946282718 learned_addr learned my addr 172.21.15.106:0/3946282718 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:39:18.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.636+0000 7f196a554640 10 monclient(hunting): get_auth_request con 0x7f1964146090 auth_method 0 2024-04-04T21:39:18.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.636+0000 7f196a554640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:18.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.636+0000 7f196a554640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.636+0000 7f196a554640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.636+0000 7f1969552640 10 monclient(hunting): get_auth_request con 0x7f196413fc70 auth_method 0 2024-04-04T21:39:18.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.636+0000 7f1969552640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:18.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.636+0000 7f1969552640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.636+0000 7f1969552640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.638 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.636+0000 7f1969552640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:18.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.636+0000 7f1969552640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:18.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.636+0000 7f1969552640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:18.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.636+0000 7f196a554640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:18.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.636+0000 7f196a554640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:18.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.636+0000 7f196a554640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:18.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.638+0000 7f1969552640 10 monclient(hunting): handle_auth_done global_id 17684 payload 274 2024-04-04T21:39:18.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.638+0000 7f1969552640 1 -- 172.21.15.106:0/3946282718 >> v2:172.21.15.106:3300/0 conn(0x7f1964146090 msgr2=0x7f196414cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:18.640 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.638+0000 7f1969552640 1 --2- 172.21.15.106:0/3946282718 >> v2:172.21.15.106:3300/0 conn(0x7f1964146090 0x7f196414cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.640 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.638+0000 7f1969552640 1 -- 172.21.15.106:0/3946282718 >> v1:172.21.15.203:6789/0 conn(0x7f19640b5360 legacy=0x7f196413f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.640 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.638+0000 7f1969552640 10 monclient: _finish_hunting 0 2024-04-04T21:39:18.640 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.638+0000 7f1969552640 1 monclient: found mon.noname-b 2024-04-04T21:39:18.640 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.638+0000 7f1969552640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:18.640 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.638+0000 7f1969552640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:39:18.640 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.638+0000 7f1969552640 1 -- 172.21.15.106:0/3946282718 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19640ab6c0 con 0x7f196413fc70 2024-04-04T21:39:18.640 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.640+0000 7f1969552640 1 --2- 172.21.15.106:0/3946282718 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f196413fc70 0x7f1964145b50 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7f19540097a0 tx=0x7f195402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a39574a1c3e5102b server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:18.644 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f1968d51640 1 -- 172.21.15.106:0/3946282718 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954038070 con 0x7f196413fc70 2024-04-04T21:39:18.644 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f1968d51640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:18.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f1968d51640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:18.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f1968d51640 10 monclient: dump: 2024-04-04T21:39:18.645 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:18.645 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.645 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.645 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.645 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:18.645 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.645 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:18.645 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:18.645 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:18.645 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f1968d51640 10 monclient: _finish_auth 0 2024-04-04T21:39:18.645 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f1968d51640 10 monclient: _check_auth_tickets 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f1968d51640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f1968d51640 1 -- 172.21.15.106:0/3946282718 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1954031700 con 0x7f196413fc70 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f1968d51640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f1968d51640 1 -- 172.21.15.106:0/3946282718 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954031ad0 con 0x7f196413fc70 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f196bfb7640 5 monclient: authenticate success, global_id 17684 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f196bfb7640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f196bfb7640 4 set_mon_vals no callback set 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f1968d51640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f1968d51640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.644+0000 7f1968d51640 10 monclient: dump: 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.646 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:18.647 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.647 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:18.647 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:18.647 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:18.647 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.645+0000 7f196bfb7640 10 monclient: shutdown 2024-04-04T21:39:18.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.645+0000 7f196bfb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:18.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.645+0000 7f196bfb7640 1 -- 172.21.15.106:0/3946282718 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f196413fc70 msgr2=0x7f1964145b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.645+0000 7f196bfb7640 1 --2- 172.21.15.106:0/3946282718 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f196413fc70 0x7f1964145b50 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7f19540097a0 tx=0x7f195402b7a0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.645+0000 7f196bfb7640 1 -- 172.21.15.106:0/3946282718 shutdown_connections 2024-04-04T21:39:18.650 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.645+0000 7f196bfb7640 1 --2- 172.21.15.106:0/3946282718 >> v2:172.21.15.106:3300/0 conn(0x7f1964146090 0x7f196414cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.645+0000 7f196bfb7640 1 --2- 172.21.15.106:0/3946282718 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f196413fc70 0x7f1964145b50 unknown :-1 s=CLOSED pgs=3271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.645+0000 7f196bfb7640 1 -- 172.21.15.106:0/3946282718 >> 172.21.15.106:0/3946282718 conn(0x7f1964066e70 msgr2=0x7f1964067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.645+0000 7f196bfb7640 1 -- 172.21.15.106:0/3946282718 shutdown_connections 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.645+0000 7f196bfb7640 1 -- 172.21.15.106:0/3946282718 wait complete. 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.646+0000 7f196bfb7640 10 monclient: build_initial_monmap 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.646+0000 7f196bfb7640 10 monclient: monmap: 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:18.651 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.646+0000 7f196bfb7640 1 librados: starting msgr at 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.646+0000 7f196bfb7640 1 librados: starting objecter 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.646+0000 7f196bfb7640 1 Processor -- start 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.646+0000 7f196bfb7640 1 -- start start 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.646+0000 7f196bfb7640 1 librados: setting wanted keys 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.646+0000 7f196bfb7640 1 librados: calling monclient init 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.646+0000 7f196bfb7640 10 monclient: init 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19640b5360 0x7f19641cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 10 monclient(hunting): picked mon.noname-a con 0x7f19640b5360 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f196413fc70 0x7f19641cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 10 monclient(hunting): picked mon.noname-c con 0x7f196413fc70 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:18.652 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1964146090 0x7f19641ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 10 monclient(hunting): picked mon.noname-b con 0x7f1964146090 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1964140110 con 0x7f19640b5360 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1964146520 con 0x7f1964146090 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19640b2650 con 0x7f196413fc70 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f196bfb7640 10 monclient(hunting): authenticate will time out at 3607.914679s 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.647+0000 7f1969d53640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19640b5360 0x7f19641cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f1969552640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f196413fc70 0x7f19641cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f196a554640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1964146090 0x7f19641ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f1969552640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f196413fc70 0x7f19641cc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:38198/0 (socket says 172.21.15.106:38198) 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f1969552640 1 -- 172.21.15.106:0/586266522 learned_addr learned my addr 172.21.15.106:0/586266522 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:18.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f1969552640 10 monclient(hunting): get_auth_request con 0x7f196413fc70 auth_method 0 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f1969552640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f1969552640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f1969552640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f1969d53640 10 monclient(hunting): get_auth_request con 0x7f19640b5360 auth_method 0 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f1969d53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f1969d53640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f1969d53640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f196a554640 10 monclient(hunting): get_auth_request con 0x7f1964146090 auth_method 0 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f196a554640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f196a554640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f196a554640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f1969552640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f1969552640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f1969552640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:18.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f196a554640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:18.655 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f196a554640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:18.655 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.648+0000 7f196a554640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:18.655 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.649+0000 7f1969d53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:18.655 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.649+0000 7f1969d53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:18.655 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.649+0000 7f1969d53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:18.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.651+0000 7f196a554640 10 monclient(hunting): handle_auth_done global_id 22114 payload 824 2024-04-04T21:39:18.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.651+0000 7f196a554640 1 -- 172.21.15.106:0/586266522 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f196413fc70 msgr2=0x7f19641cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.651+0000 7f196a554640 1 --2- 172.21.15.106:0/586266522 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f196413fc70 0x7f19641cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.651+0000 7f196a554640 1 -- 172.21.15.106:0/586266522 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19640b5360 msgr2=0x7f19641cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.651+0000 7f196a554640 1 --2- 172.21.15.106:0/586266522 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19640b5360 0x7f19641cf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.651+0000 7f196a554640 10 monclient: _finish_hunting 0 2024-04-04T21:39:18.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.651+0000 7f196a554640 1 monclient: found mon.noname-b 2024-04-04T21:39:18.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.651+0000 7f196a554640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:18.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.651+0000 7f196a554640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:18.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.651+0000 7f196a554640 1 -- 172.21.15.106:0/586266522 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19640ac5a0 con 0x7f1964146090 2024-04-04T21:39:18.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.655+0000 7f196a554640 1 --2- 172.21.15.106:0/586266522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1964146090 0x7f19641ccf00 secure :-1 s=READY pgs=5551 cs=0 l=1 rev1=1 crypto rx=0x7f195c00eb80 tx=0x7f195c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:18.656 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.655+0000 7f195a7fc640 10 client.?.objecter ms_handle_connect 0x7f1964146090 2024-04-04T21:39:18.657 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.655+0000 7f195a7fc640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:18.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f195a7fc640 1 -- 172.21.15.106:0/586266522 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c018020 con 0x7f1964146090 2024-04-04T21:39:18.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f195a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:18.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f195a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:18.660 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f195a7fc640 10 monclient: dump: 2024-04-04T21:39:18.660 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f195a7fc640 10 monclient: _finish_auth 0 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f195a7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f195a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f195a7fc640 1 -- 172.21.15.106:0/586266522 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195c00cd40 con 0x7f1964146090 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f195a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:18.661 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f195a7fc640 1 -- 172.21.15.106:0/586266522 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c0146c0 con 0x7f1964146090 2024-04-04T21:39:18.662 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f196bfb7640 5 monclient: authenticate success, global_id 22114 2024-04-04T21:39:18.662 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f196bfb7640 10 monclient: _renew_subs 2024-04-04T21:39:18.662 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f196bfb7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:18.662 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f196bfb7640 1 -- 172.21.15.106:0/586266522 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19640ad170 con 0x7f1964146090 2024-04-04T21:39:18.662 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f195a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:18.662 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f195bfff640 4 set_mon_vals no callback set 2024-04-04T21:39:18.662 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f195a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:18.662 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.659+0000 7f195a7fc640 10 monclient: dump: 2024-04-04T21:39:18.662 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:18.662 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.662 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.662 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.662 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:18.662 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.660+0000 7f196bfb7640 10 client.22114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.660+0000 7f196bfb7640 10 monclient: _renew_subs 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.660+0000 7f196bfb7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.660+0000 7f196bfb7640 1 -- 172.21.15.106:0/586266522 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19640ac080 con 0x7f1964146090 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.660+0000 7f196bfb7640 1 librados: init done 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.660+0000 7f196bfb7640 10 client.22114.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.660+0000 7f196bfb7640 15 client.22114.objecter _session_command_op_assign -1 1 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.660+0000 7f196bfb7640 20 client.22114.objecter _get_session osd=-1 returning homeless 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.660+0000 7f196bfb7640 10 client.22114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.660+0000 7f196bfb7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.660+0000 7f196bfb7640 1 -- 172.21.15.106:0/586266522 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1964069c40 con 0x7f1964146090 2024-04-04T21:39:18.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.680+0000 7f195a7fc640 1 -- 172.21.15.106:0/586266522 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f195c005d10 con 0x7f1964146090 2024-04-04T21:39:18.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.680+0000 7f195a7fc640 1 --2- 172.21.15.106:0/586266522 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f194806ceb0 0x7f194806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.681+0000 7f1969d53640 1 --2- 172.21.15.106:0/586266522 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f194806ceb0 0x7f194806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.681+0000 7f1969d53640 10 monclient: get_auth_request con 0x7f194806ceb0 auth_method 0 2024-04-04T21:39:18.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.681+0000 7f1969d53640 1 -- 172.21.15.106:0/586266522 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f194806ceb0 msgr2=0x7f194806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-04T21:39:18.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.681+0000 7f1969d53640 1 -- 172.21.15.106:0/586266522 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f194806ceb0 msgr2=0x7f194806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-04T21:39:18.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.681+0000 7f1969d53640 1 --2- 172.21.15.106:0/586266522 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f194806ceb0 0x7f194806f370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f1960007fd0 tx=0x7f1960007e80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-04T21:39:18.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.681+0000 7f1969d53640 1 --2- 172.21.15.106:0/586266522 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f194806ceb0 0x7f194806f370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-04T21:39:18.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.683+0000 7f195a7fc640 1 -- 172.21.15.106:0/586266522 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f195c057ee0 con 0x7f1964146090 2024-04-04T21:39:18.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.683+0000 7f195a7fc640 10 client.22114.objecter ms_dispatch 0x7f1964000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:18.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.683+0000 7f195a7fc640 3 client.22114.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:18.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.683+0000 7f195a7fc640 3 client.22114.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:18.684 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.684+0000 7f195a7fc640 10 client.22114.objecter checking command 1 2024-04-04T21:39:18.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.684+0000 7f195a7fc640 1 --2- 172.21.15.106:0/586266522 >> [v2:172.21.15.203:6800/696438187,v1:172.21.15.203:6801/696438187] conn(0x7f1948074ba0 0x7f1948077040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.684+0000 7f195a7fc640 20 client.22114.objecter _get_session s=0x7f19480744d0 osd=5 3 2024-04-04T21:39:18.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.684+0000 7f195a7fc640 20 client.22114.objecter put_session s=0x7f19480744d0 osd=5 3 2024-04-04T21:39:18.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.684+0000 7f195a7fc640 15 client.22114.objecter _session_command_op_remove -1 1 2024-04-04T21:39:18.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.684+0000 7f195a7fc640 20 client.22114.objecter _get_session s=0x7f19480744d0 osd=5 3 2024-04-04T21:39:18.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.684+0000 7f195a7fc640 20 client.22114.objecter get_session s=0x7f19480744d0 osd=5 3 2024-04-04T21:39:18.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.684+0000 7f195a7fc640 15 client.22114.objecter _session_command_op_assign 5 1 2024-04-04T21:39:18.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.684+0000 7f195a7fc640 20 client.22114.objecter put_session s=0x7f19480744d0 osd=5 4 2024-04-04T21:39:18.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.684+0000 7f195a7fc640 10 client.22114.objecter _send_command 1 2024-04-04T21:39:18.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.684+0000 7f195a7fc640 1 -- 172.21.15.106:0/586266522 --> [v2:172.21.15.203:6800/696438187,v1:172.21.15.203:6801/696438187] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1948077710 con 0x7f1948074ba0 2024-04-04T21:39:18.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.684+0000 7f195a7fc640 20 client.22114.objecter dump_active .. 0 homeless 2024-04-04T21:39:18.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.685+0000 7f195a7fc640 1 -- 172.21.15.106:0/586266522 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=1197) v2 ==== 24+0+0 (secure 0 0 0) 0x7f195c0560e0 con 0x7f1964146090 2024-04-04T21:39:18.686 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.685+0000 7f195a7fc640 10 monclient: handle_get_version_reply finishing 1 version 1197 2024-04-04T21:39:18.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.711+0000 7f1969552640 1 --2- 172.21.15.106:0/586266522 >> [v2:172.21.15.203:6800/696438187,v1:172.21.15.203:6801/696438187] conn(0x7f1948074ba0 0x7f1948077040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.712+0000 7f1969552640 10 monclient: get_auth_request con 0x7f1948074ba0 auth_method 0 2024-04-04T21:39:18.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.717+0000 7f1969552640 1 --2- 172.21.15.106:0/586266522 >> [v2:172.21.15.203:6800/696438187,v1:172.21.15.203:6801/696438187] conn(0x7f1948074ba0 0x7f1948077040 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:18.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.717+0000 7f195a7fc640 10 client.22114.objecter ms_handle_connect 0x7f1948074ba0 2024-04-04T21:39:18.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.741+0000 7f195a7fc640 1 -- 172.21.15.106:0/586266522 <== osd.5 v2:172.21.15.203:6800/696438187 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1948077710 con 0x7f1948074ba0 2024-04-04T21:39:18.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.741+0000 7f195a7fc640 10 client.22114.objecter ms_dispatch 0x7f1964000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:39:18.741 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.741+0000 7f195a7fc640 10 client.22114.objecter _finish_command 1 = system:0 2024-04-04T21:39:18.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.741+0000 7f195a7fc640 20 client.22114.objecter put_session s=0x7f19480744d0 osd=5 4 2024-04-04T21:39:18.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.741+0000 7f195a7fc640 15 client.22114.objecter _session_command_op_remove 5 1 2024-04-04T21:39:18.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.754+0000 7f196bfb7640 10 client.22114.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-04T21:39:18.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.754+0000 7f196bfb7640 15 client.22114.objecter _session_command_op_assign -1 2 2024-04-04T21:39:18.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.754+0000 7f196bfb7640 20 client.22114.objecter _get_session s=0x7f19480744d0 osd=5 3 2024-04-04T21:39:18.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.754+0000 7f196bfb7640 20 client.22114.objecter put_session s=0x7f19480744d0 osd=5 3 2024-04-04T21:39:18.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.754+0000 7f196bfb7640 20 client.22114.objecter _get_session s=0x7f19480744d0 osd=5 3 2024-04-04T21:39:18.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.754+0000 7f196bfb7640 15 client.22114.objecter _session_command_op_remove -1 2 2024-04-04T21:39:18.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.754+0000 7f196bfb7640 20 client.22114.objecter get_session s=0x7f19480744d0 osd=5 3 2024-04-04T21:39:18.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.754+0000 7f196bfb7640 15 client.22114.objecter _session_command_op_assign 5 2 2024-04-04T21:39:18.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.754+0000 7f196bfb7640 20 client.22114.objecter put_session s=0x7f19480744d0 osd=5 4 2024-04-04T21:39:18.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.754+0000 7f196bfb7640 10 client.22114.objecter _send_command 2 2024-04-04T21:39:18.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.754+0000 7f196bfb7640 1 -- 172.21.15.106:0/586266522 --> [v2:172.21.15.203:6800/696438187,v1:172.21.15.203:6801/696438187] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f196413cca0 con 0x7f1948074ba0 2024-04-04T21:39:18.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.770+0000 7f195a7fc640 1 -- 172.21.15.106:0/586266522 <== osd.5 v2:172.21.15.203:6800/696438187 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f196413cca0 con 0x7f1948074ba0 2024-04-04T21:39:18.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.770+0000 7f195a7fc640 10 client.22114.objecter ms_dispatch 0x7f1964000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:39:18.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.770+0000 7f195a7fc640 10 client.22114.objecter _finish_command 2 = system:0 2024-04-04T21:39:18.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.770+0000 7f195a7fc640 20 client.22114.objecter put_session s=0x7f19480744d0 osd=5 4 2024-04-04T21:39:18.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.770+0000 7f195a7fc640 15 client.22114.objecter _session_command_op_remove 5 2 2024-04-04T21:39:18.772 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-04T21:39:18.772 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-04T21:39:18.772 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-04T21:39:18.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 10 librados: watch_flush enter 2024-04-04T21:39:18.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 10 librados: watch_flush exit 2024-04-04T21:39:18.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 10 client.22114.objecter close_session for osd.5 2024-04-04T21:39:18.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 1 -- 172.21.15.106:0/586266522 >> [v2:172.21.15.203:6800/696438187,v1:172.21.15.203:6801/696438187] conn(0x7f1948074ba0 msgr2=0x7f1948077040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 1 --2- 172.21.15.106:0/586266522 >> [v2:172.21.15.203:6800/696438187,v1:172.21.15.203:6801/696438187] conn(0x7f1948074ba0 0x7f1948077040 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 20 client.22114.objecter put_session s=0x7f19480744d0 osd=5 1 2024-04-04T21:39:18.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 20 client.22114.objecter shutdown clearing up homeless session... 2024-04-04T21:39:18.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 10 client.22114.objecter successfully canceled tick 2024-04-04T21:39:18.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 1 -- 172.21.15.106:0/586266522 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f194806ceb0 msgr2=0x7f194806f370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-04T21:39:18.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 1 --2- 172.21.15.106:0/586266522 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f194806ceb0 0x7f194806f370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 10 monclient: shutdown 2024-04-04T21:39:18.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:18.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 1 -- 172.21.15.106:0/586266522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1964146090 msgr2=0x7f19641ccf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 1 --2- 172.21.15.106:0/586266522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1964146090 0x7f19641ccf00 secure :-1 s=READY pgs=5551 cs=0 l=1 rev1=1 crypto rx=0x7f195c00eb80 tx=0x7f195c00c490 comp rx=0 tx=0).stop 2024-04-04T21:39:18.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196a554640 1 -- 172.21.15.106:0/586266522 reap_dead start 2024-04-04T21:39:18.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 1 -- 172.21.15.106:0/586266522 shutdown_connections 2024-04-04T21:39:18.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 1 -- 172.21.15.106:0/586266522 >> 172.21.15.106:0/586266522 conn(0x7f1964066e70 msgr2=0x7f1964145790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:18.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 1 -- 172.21.15.106:0/586266522 shutdown_connections 2024-04-04T21:39:18.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 1 -- 172.21.15.106:0/586266522 wait complete. 2024-04-04T21:39:18.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.771+0000 7f196bfb7640 1 librados: shutdown 2024-04-04T21:39:18.782 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-04T21:39:18.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.885+0000 7fcacbe0c640 10 monclient: get_monmap_and_config 2024-04-04T21:39:18.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.885+0000 7fcacbe0c640 10 monclient: build_initial_monmap 2024-04-04T21:39:18.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.885+0000 7fcacbe0c640 10 monclient: monmap: 2024-04-04T21:39:18.887 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:18.887 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.887 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:18.886674+0000 2024-04-04T21:39:18.887 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:18.886674+0000 2024-04-04T21:39:18.887 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:18.887 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.888 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:18.888 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:18.888 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:18.888 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:18.888 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:18.888 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 1 Processor -- start 2024-04-04T21:39:18.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 1 -- start start 2024-04-04T21:39:18.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 10 monclient: init 2024-04-04T21:39:18.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:18.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-04-04T21:39:18.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcac40b5200 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:18.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcac40a9aa0 0x7fcac4145af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 10 monclient(hunting): picked mon.noname-b con 0x7fcac40a9aa0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:18.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcac4146030 addr v1:172.21.15.203:6789/0 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac40a97e0 con 0x7fcac40b5200 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac40ab420 con 0x7fcac4146030 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac40b55e0 con 0x7fcac40a9aa0 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.886+0000 7fcacbe0c640 10 monclient(hunting): authenticate will time out at 3608.154179s 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.887+0000 7fcac93a7640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcac40a9aa0 0x7fcac4145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.887+0000 7fcac9ba8640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fcac40b5200 0x7fcac413f9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:43872/0 (socket says 172.21.15.106:43872) 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.887+0000 7fcac9ba8640 1 -- 172.21.15.106:0/2131428566 learned_addr learned my addr 172.21.15.106:0/2131428566 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.887+0000 7fcac93a7640 10 monclient(hunting): get_auth_request con 0x7fcac40a9aa0 auth_method 0 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.887+0000 7fcac93a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.887+0000 7fcac93a7640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.887+0000 7fcac93a7640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.888+0000 7fcac93a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:18.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.888+0000 7fcac93a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.888+0000 7fcac93a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.889+0000 7fcac8ba6640 1 -- 172.21.15.106:0/2131428566 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226287022 0 0) 0x7fcac40a97e0 con 0x7fcac40b5200 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.889+0000 7fcac8ba6640 10 monclient(hunting): _negotiate 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.889+0000 7fcac8ba6640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.889+0000 7fcac8ba6640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac8ba6640 10 monclient(hunting): my global_id is 15153 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac8ba6640 1 -- 172.21.15.106:0/2131428566 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcab0003520 con 0x7fcac40b5200 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac8ba6640 1 -- 172.21.15.106:0/2131428566 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458066413 0 0) 0x7fcac40ab420 con 0x7fcac4146030 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac8ba6640 10 monclient(hunting): _negotiate 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac8ba6640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac8ba6640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac8ba6640 10 monclient(hunting): my global_id is 22117 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac8ba6640 1 -- 172.21.15.106:0/2131428566 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac40a97e0 con 0x7fcac4146030 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac93a7640 10 monclient(hunting): handle_auth_done global_id 17690 payload 274 2024-04-04T21:39:18.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac93a7640 1 -- 172.21.15.106:0/2131428566 >> v1:172.21.15.203:6789/0 conn(0x7fcac4146030 legacy=0x7fcac414ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac93a7640 1 -- 172.21.15.106:0/2131428566 >> v1:172.21.15.106:6789/0 conn(0x7fcac40b5200 legacy=0x7fcac413f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac93a7640 10 monclient: _finish_hunting 0 2024-04-04T21:39:18.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac93a7640 1 monclient: found mon.noname-b 2024-04-04T21:39:18.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac93a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:18.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac93a7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:39:18.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.890+0000 7fcac93a7640 1 -- 172.21.15.106:0/2131428566 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac40ab700 con 0x7fcac40a9aa0 2024-04-04T21:39:18.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.891+0000 7fcac93a7640 1 --2- 172.21.15.106:0/2131428566 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcac40a9aa0 0x7fcac4145af0 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7fcab40097d0 tx=0x7fcab402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12aac22ac6bac52 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:18.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcac8ba6640 1 -- 172.21.15.106:0/2131428566 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab4035020 con 0x7fcac40a9aa0 2024-04-04T21:39:18.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcac8ba6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcac8ba6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcac8ba6640 10 monclient: dump: 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcac8ba6640 10 monclient: _finish_auth 0 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcac8ba6640 10 monclient: _check_auth_tickets 2024-04-04T21:39:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcac8ba6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:18.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcacbe0c640 5 monclient: authenticate success, global_id 17690 2024-04-04T21:39:18.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcac8ba6640 1 -- 172.21.15.106:0/2131428566 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab4031800 con 0x7fcac40a9aa0 2024-04-04T21:39:18.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcacbe0c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:39:18.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcac8ba6640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:18.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcacbe0c640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:18.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcacbe0c640 4 set_mon_vals no callback set 2024-04-04T21:39:18.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcac8ba6640 1 -- 172.21.15.106:0/2131428566 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab4031bd0 con 0x7fcac40a9aa0 2024-04-04T21:39:18.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcac8ba6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:18.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcac8ba6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:39:18.898 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcac8ba6640 10 monclient: dump: 2024-04-04T21:39:18.898 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:18.898 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.898 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.898 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcacbe0c640 10 monclient: shutdown 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcacbe0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcacbe0c640 1 -- 172.21.15.106:0/2131428566 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcac40a9aa0 msgr2=0x7fcac4145af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcacbe0c640 1 --2- 172.21.15.106:0/2131428566 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcac40a9aa0 0x7fcac4145af0 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7fcab40097d0 tx=0x7fcab402b8a0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcacbe0c640 1 -- 172.21.15.106:0/2131428566 shutdown_connections 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcacbe0c640 1 --2- 172.21.15.106:0/2131428566 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcac40a9aa0 0x7fcac4145af0 unknown :-1 s=CLOSED pgs=3272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.895+0000 7fcacbe0c640 1 -- 172.21.15.106:0/2131428566 >> 172.21.15.106:0/2131428566 conn(0x7fcac4066e40 msgr2=0x7fcac4067250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.896+0000 7fcacbe0c640 1 -- 172.21.15.106:0/2131428566 shutdown_connections 2024-04-04T21:39:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.896+0000 7fcacbe0c640 1 -- 172.21.15.106:0/2131428566 wait complete. 2024-04-04T21:39:18.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.896+0000 7fcacbe0c640 10 monclient: build_initial_monmap 2024-04-04T21:39:18.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.896+0000 7fcacbe0c640 10 monclient: monmap: 2024-04-04T21:39:18.900 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:18.900 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:18.900 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:18.900 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:18.900 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:18.900 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.900 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:18.901 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:18.901 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:18.901 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.896+0000 7fcacbe0c640 1 librados: starting msgr at 2024-04-04T21:39:18.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.896+0000 7fcacbe0c640 1 librados: starting objecter 2024-04-04T21:39:18.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.896+0000 7fcacbe0c640 1 Processor -- start 2024-04-04T21:39:18.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 1 -- start start 2024-04-04T21:39:18.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 1 librados: setting wanted keys 2024-04-04T21:39:18.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 1 librados: calling monclient init 2024-04-04T21:39:18.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 10 monclient: init 2024-04-04T21:39:18.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:18.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-04T21:39:18.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcac40a9aa0 0x7fcac4140e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 10 monclient(hunting): picked mon.noname-b con 0x7fcac40a9aa0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:18.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcac40b5200 0x7fcac4141a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 10 monclient(hunting): picked mon.noname-c con 0x7fcac40b5200 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcac4146030 0x7fcac41d7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 10 monclient(hunting): picked mon.noname-a con 0x7fcac4146030 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac4140240 con 0x7fcac4146030 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac41400c0 con 0x7fcac40a9aa0 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac40aa500 con 0x7fcac40b5200 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcacbe0c640 10 monclient(hunting): authenticate will time out at 3608.164699s 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcac93a7640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcac40b5200 0x7fcac4141a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.897+0000 7fcaca3a9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcac4146030 0x7fcac41d7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcac93a7640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcac40b5200 0x7fcac4141a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:38222/0 (socket says 172.21.15.106:38222) 2024-04-04T21:39:18.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcac9ba8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcac40a9aa0 0x7fcac4140e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:18.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcac93a7640 1 -- 172.21.15.106:0/1797265571 learned_addr learned my addr 172.21.15.106:0/1797265571 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:18.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcac93a7640 10 monclient(hunting): get_auth_request con 0x7fcac40b5200 auth_method 0 2024-04-04T21:39:18.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcac93a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:18.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcac93a7640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcac93a7640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcaca3a9640 10 monclient(hunting): get_auth_request con 0x7fcac4146030 auth_method 0 2024-04-04T21:39:18.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcaca3a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:18.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcaca3a9640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcaca3a9640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcac9ba8640 10 monclient(hunting): get_auth_request con 0x7fcac40a9aa0 auth_method 0 2024-04-04T21:39:18.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcac9ba8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:18.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcac9ba8640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:18.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcac9ba8640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:18.906 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcac93a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcac93a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcac93a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcaca3a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcaca3a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.898+0000 7fcaca3a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.899+0000 7fcac9ba8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.899+0000 7fcac9ba8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.899+0000 7fcac9ba8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.901+0000 7fcac93a7640 10 monclient(hunting): handle_auth_done global_id 17693 payload 824 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.901+0000 7fcac93a7640 1 -- 172.21.15.106:0/1797265571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcac40a9aa0 msgr2=0x7fcac4140e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.901+0000 7fcac93a7640 1 --2- 172.21.15.106:0/1797265571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcac40a9aa0 0x7fcac4140e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.901+0000 7fcac93a7640 1 -- 172.21.15.106:0/1797265571 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcac4146030 msgr2=0x7fcac41d7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.901+0000 7fcac93a7640 1 --2- 172.21.15.106:0/1797265571 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcac4146030 0x7fcac41d7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.901+0000 7fcac93a7640 10 monclient: _finish_hunting 0 2024-04-04T21:39:18.907 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.901+0000 7fcac93a7640 1 monclient: found mon.noname-c 2024-04-04T21:39:18.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.901+0000 7fcac93a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:18.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.901+0000 7fcac93a7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:39:18.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.901+0000 7fcac93a7640 1 -- 172.21.15.106:0/1797265571 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac4143250 con 0x7fcac40b5200 2024-04-04T21:39:18.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.903+0000 7fcac93a7640 1 --2- 172.21.15.106:0/1797265571 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcac40b5200 0x7fcac4141a50 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7fcab40098f0 tx=0x7fcab40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:18.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.903+0000 7fcaba7fc640 10 client.?.objecter ms_handle_connect 0x7fcac40b5200 2024-04-04T21:39:18.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.903+0000 7fcaba7fc640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:18.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcaba7fc640 1 -- 172.21.15.106:0/1797265571 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab4041070 con 0x7fcac40b5200 2024-04-04T21:39:18.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcaba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:18.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcaba7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:39:18.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcaba7fc640 10 monclient: dump: 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcaba7fc640 10 monclient: _finish_auth 0 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcaba7fc640 10 monclient: _check_auth_tickets 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcaba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcacbe0c640 5 monclient: authenticate success, global_id 17693 2024-04-04T21:39:18.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcacbe0c640 10 monclient: _renew_subs 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcacbe0c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcacbe0c640 1 -- 172.21.15.106:0/1797265571 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac4143a00 con 0x7fcac40b5200 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcacbe0c640 10 client.17693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcacbe0c640 10 monclient: _renew_subs 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcacbe0c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcacbe0c640 1 -- 172.21.15.106:0/1797265571 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac4142170 con 0x7fcac40b5200 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.905+0000 7fcacbe0c640 1 librados: init done 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.906+0000 7fcacbe0c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.906+0000 7fcacbe0c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.906+0000 7fcacbe0c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.906+0000 7fcacbe0c640 1 -- 172.21.15.106:0/1797265571 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac404f050 con 0x7fcac40b5200 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.908+0000 7fcaba7fc640 1 -- 172.21.15.106:0/1797265571 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab4031800 con 0x7fcac40b5200 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.908+0000 7fcaba7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.908+0000 7fcaba7fc640 1 -- 172.21.15.106:0/1797265571 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab4035040 con 0x7fcac40b5200 2024-04-04T21:39:18.911 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.908+0000 7fcaba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:18.912 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.908+0000 7fcaba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:39:18.912 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.908+0000 7fcaba7fc640 10 monclient: dump: 2024-04-04T21:39:18.912 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:18.912 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:18.912 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.912 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:18.912 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:18.912 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:18.912 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:18.912 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:18.912 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:18.912 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:18.912 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:18.908+0000 7fcabb7fe640 4 set_mon_vals no callback set 2024-04-04T21:39:19.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.149+0000 7fcaba7fc640 1 -- 172.21.15.106:0/1797265571 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fcab404c060 con 0x7fcac40b5200 2024-04-04T21:39:19.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.149+0000 7fcaba7fc640 1 --2- 172.21.15.106:0/1797265571 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fcaa806ce60 0x7fcaa806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:19.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.149+0000 7fcac9ba8640 1 --2- 172.21.15.106:0/1797265571 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fcaa806ce60 0x7fcaa806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:19.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.149+0000 7fcac9ba8640 10 monclient: get_auth_request con 0x7fcaa806ce60 auth_method 0 2024-04-04T21:39:19.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.150+0000 7fcaba7fc640 10 client.17693.objecter ms_handle_connect 0x7fcaa806ce60 2024-04-04T21:39:19.151 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.150+0000 7fcac9ba8640 1 --2- 172.21.15.106:0/1797265571 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fcaa806ce60 0x7fcaa806f320 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcac0000f60 tx=0x7fcac0007d30 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:19.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.162+0000 7fcaba7fc640 1 -- 172.21.15.106:0/1797265571 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7fcab4078870 con 0x7fcac40b5200 2024-04-04T21:39:19.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.162+0000 7fcaba7fc640 10 client.17693.objecter ms_dispatch 0x7fcac4000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:19.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.162+0000 7fcaba7fc640 3 client.17693.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:19.162 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.162+0000 7fcaba7fc640 3 client.17693.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:19.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.162+0000 7fcaba7fc640 20 client.17693.objecter dump_active .. 0 homeless 2024-04-04T21:39:19.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.162+0000 7fcaba7fc640 1 -- 172.21.15.106:0/1797265571 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcab407a7d0 con 0x7fcac40b5200 2024-04-04T21:39:19.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.162+0000 7fcaba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:19.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.162+0000 7fcaba7fc640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:19.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.352+0000 7fcacbe0c640 1 -- 172.21.15.106:0/1797265571 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fcac41d7780 con 0x7fcaa806ce60 2024-04-04T21:39:19.353 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.352+0000 7fcaba7fc640 1 -- 172.21.15.106:0/1797265571 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcac41d7780 con 0x7fcaa806ce60 2024-04-04T21:39:19.353 INFO:teuthology.orchestra.run.smithi106.stderr:instructed osd(s) 5 to deep-scrub 2024-04-04T21:39:19.355 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.355+0000 7fcacbe0c640 10 librados: watch_flush enter 2024-04-04T21:39:19.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.355+0000 7fcacbe0c640 10 librados: watch_flush exit 2024-04-04T21:39:19.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.355+0000 7fcacbe0c640 20 client.17693.objecter shutdown clearing up homeless session... 2024-04-04T21:39:19.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.355+0000 7fcacbe0c640 10 client.17693.objecter successfully canceled tick 2024-04-04T21:39:19.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.356+0000 7fcacbe0c640 1 -- 172.21.15.106:0/1797265571 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fcaa806ce60 msgr2=0x7fcaa806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:19.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.356+0000 7fcacbe0c640 1 --2- 172.21.15.106:0/1797265571 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fcaa806ce60 0x7fcaa806f320 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcac0000f60 tx=0x7fcac0007d30 comp rx=0 tx=0).stop 2024-04-04T21:39:19.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.356+0000 7fcacbe0c640 10 monclient: shutdown 2024-04-04T21:39:19.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.356+0000 7fcacbe0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:19.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.356+0000 7fcacbe0c640 1 -- 172.21.15.106:0/1797265571 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcac40b5200 msgr2=0x7fcac4141a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:19.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.356+0000 7fcacbe0c640 1 --2- 172.21.15.106:0/1797265571 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcac40b5200 0x7fcac4141a50 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7fcab40098f0 tx=0x7fcab40046c0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.356+0000 7fcacbe0c640 1 -- 172.21.15.106:0/1797265571 shutdown_connections 2024-04-04T21:39:19.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.356+0000 7fcacbe0c640 1 --2- 172.21.15.106:0/1797265571 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fcaa806ce60 0x7fcaa806f320 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.356+0000 7fcacbe0c640 1 --2- 172.21.15.106:0/1797265571 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcac4146030 0x7fcac41d7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.356+0000 7fcacbe0c640 1 --2- 172.21.15.106:0/1797265571 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcac40b5200 0x7fcac4141a50 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.356+0000 7fcacbe0c640 1 --2- 172.21.15.106:0/1797265571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcac40a9aa0 0x7fcac4140e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.356+0000 7fcacbe0c640 1 -- 172.21.15.106:0/1797265571 >> 172.21.15.106:0/1797265571 conn(0x7fcac4066e40 msgr2=0x7fcac40b7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:19.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.357+0000 7fcacbe0c640 1 -- 172.21.15.106:0/1797265571 shutdown_connections 2024-04-04T21:39:19.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.357+0000 7fcacbe0c640 1 -- 172.21.15.106:0/1797265571 wait complete. 2024-04-04T21:39:19.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.357+0000 7fcacbe0c640 1 librados: shutdown 2024-04-04T21:39:19.371 INFO:tasks.ceph:Scrubbing osd.6 2024-04-04T21:39:19.371 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-04T21:39:19.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.485+0000 7fbaf46ee640 10 monclient: get_monmap_and_config 2024-04-04T21:39:19.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.486+0000 7fbaf46ee640 10 monclient: build_initial_monmap 2024-04-04T21:39:19.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.486+0000 7fbaf46ee640 10 monclient: monmap: 2024-04-04T21:39:19.486 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:19.486 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:19.486 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:19.487081+0000 2024-04-04T21:39:19.486 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:19.487081+0000 2024-04-04T21:39:19.486 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:19.487 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:19.487 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:19.487 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:19.487 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:19.487 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:19.487 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:19.487 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.486+0000 7fbaf46ee640 1 Processor -- start 2024-04-04T21:39:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.486+0000 7fbaf46ee640 1 -- start start 2024-04-04T21:39:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.486+0000 7fbaf46ee640 10 monclient: init 2024-04-04T21:39:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf46ee640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf46ee640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-04-04T21:39:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf46ee640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbaec143100 addr v1:172.21.15.203:6789/0 2024-04-04T21:39:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf46ee640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbaec0b1bf0 0x7fbaec0b2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf46ee640 10 monclient(hunting): picked mon.noname-a con 0x7fbaec0b1bf0 addr v2:172.21.15.106:3300/0 2024-04-04T21:39:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf46ee640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fbaec0afbd0 0x7fbaec0affd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf46ee640 10 monclient(hunting): picked mon.noname-c con 0x7fbaec0afbd0 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf46ee640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaec0b17d0 con 0x7fbaec143100 2024-04-04T21:39:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf46ee640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf46ee640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaec1426f0 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf46ee640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:19.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf46ee640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaec0b26e0 con 0x7fbaec0afbd0 2024-04-04T21:39:19.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf46ee640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:19.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf46ee640 10 monclient(hunting): authenticate will time out at 3608.754628s 2024-04-04T21:39:19.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.487+0000 7fbaf1c89640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbaec0b1bf0 0x7fbaec0b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:19.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.488+0000 7fbaf1c89640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbaec0b1bf0 0x7fbaec0b2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42530/0 (socket says 172.21.15.106:42530) 2024-04-04T21:39:19.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.488+0000 7fbaf1c89640 1 -- 172.21.15.106:0/3451375735 learned_addr learned my addr 172.21.15.106:0/3451375735 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:19.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.488+0000 7fbaf1c89640 10 monclient(hunting): get_auth_request con 0x7fbaec0b1bf0 auth_method 0 2024-04-04T21:39:19.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.488+0000 7fbaf1c89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:19.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.488+0000 7fbaf1c89640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:19.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.488+0000 7fbaf1c89640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:19.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.488+0000 7fbaf1c89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:19.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.488+0000 7fbaf1c89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:19.490 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.488+0000 7fbaf1c89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:19.491 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.490+0000 7fbaf1c89640 10 monclient(hunting): handle_auth_done global_id 15159 payload 274 2024-04-04T21:39:19.491 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.490+0000 7fbaf1c89640 1 -- 172.21.15.106:0/3451375735 >> v2:172.21.15.203:3300/0 conn(0x7fbaec0afbd0 msgr2=0x7fbaec0affd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:19.491 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.490+0000 7fbaf1c89640 1 --2- 172.21.15.106:0/3451375735 >> v2:172.21.15.203:3300/0 conn(0x7fbaec0afbd0 0x7fbaec0affd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.491 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.490+0000 7fbaf1c89640 1 -- 172.21.15.106:0/3451375735 >> v1:172.21.15.203:6789/0 conn(0x7fbaec143100 legacy=0x7fbaec0b5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:19.491 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.490+0000 7fbaf1c89640 10 monclient: _finish_hunting 0 2024-04-04T21:39:19.491 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.490+0000 7fbaf1c89640 1 monclient: found mon.noname-a 2024-04-04T21:39:19.491 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.490+0000 7fbaf1c89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:19.491 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.490+0000 7fbaf1c89640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:39:19.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.490+0000 7fbaf1c89640 1 -- 172.21.15.106:0/3451375735 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaec1434e0 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.492 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.492+0000 7fbaf1c89640 1 --2- 172.21.15.106:0/3451375735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaec0b1bf0 0x7fbaec0b2030 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 crypto rx=0x7fbaec0b3fe0 tx=0x7fbadc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a6a611752a5f26 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:19.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf1488640 1 -- 172.21.15.106:0/3451375735 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc02e070 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf1488640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:19.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf1488640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:39:19.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf1488640 10 monclient: dump: 2024-04-04T21:39:19.496 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:19.496 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:19.496 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.496 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf1488640 10 monclient: _finish_auth 0 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf1488640 10 monclient: _check_auth_tickets 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf1488640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf1488640 1 -- 172.21.15.106:0/3451375735 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc037070 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf1488640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf1488640 1 -- 172.21.15.106:0/3451375735 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc031a00 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf46ee640 5 monclient: authenticate success, global_id 15159 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf46ee640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:19.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf46ee640 4 set_mon_vals no callback set 2024-04-04T21:39:19.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf1488640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:19.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf1488640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:39:19.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf1488640 10 monclient: dump: 2024-04-04T21:39:19.498 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:19.498 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:19.498 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.498 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.498 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:19.498 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:19.498 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:19.498 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:19.498 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:19.498 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:19.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf46ee640 10 monclient: shutdown 2024-04-04T21:39:19.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf46ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:19.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf46ee640 1 -- 172.21.15.106:0/3451375735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaec0b1bf0 msgr2=0x7fbaec0b2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:19.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.496+0000 7fbaf46ee640 1 --2- 172.21.15.106:0/3451375735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaec0b1bf0 0x7fbaec0b2030 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 crypto rx=0x7fbaec0b3fe0 tx=0x7fbadc02b880 comp rx=0 tx=0).stop 2024-04-04T21:39:19.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.497+0000 7fbaf46ee640 1 -- 172.21.15.106:0/3451375735 shutdown_connections 2024-04-04T21:39:19.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.497+0000 7fbaf46ee640 1 --2- 172.21.15.106:0/3451375735 >> v2:172.21.15.203:3300/0 conn(0x7fbaec0afbd0 0x7fbaec0affd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.497+0000 7fbaf46ee640 1 --2- 172.21.15.106:0/3451375735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaec0b1bf0 0x7fbaec0b2030 unknown :-1 s=CLOSED pgs=3167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.497+0000 7fbaf46ee640 1 -- 172.21.15.106:0/3451375735 >> 172.21.15.106:0/3451375735 conn(0x7fbaec06d3a0 msgr2=0x7fbaec06d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.497+0000 7fbaf46ee640 1 -- 172.21.15.106:0/3451375735 shutdown_connections 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.497+0000 7fbaf46ee640 1 -- 172.21.15.106:0/3451375735 wait complete. 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.497+0000 7fbaf46ee640 10 monclient: build_initial_monmap 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.497+0000 7fbaf46ee640 10 monclient: monmap: 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:19.502 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 1 librados: starting msgr at 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 1 librados: starting objecter 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 1 Processor -- start 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 1 -- start start 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 1 librados: setting wanted keys 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 1 librados: calling monclient init 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 10 monclient: init 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbaec0afbd0 0x7fbaec13e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 10 monclient(hunting): picked mon.noname-c con 0x7fbaec0afbd0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaec0b1bf0 0x7fbaec0ab060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 10 monclient(hunting): picked mon.noname-a con 0x7fbaec0b1bf0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaec143100 0x7fbaec0ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:19.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.498+0000 7fbaf46ee640 10 monclient(hunting): picked mon.noname-b con 0x7fbaec143100 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf46ee640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf46ee640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec14cfa0 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf46ee640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf46ee640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec0b07a0 con 0x7fbaec143100 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf46ee640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf46ee640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec14d120 con 0x7fbaec0afbd0 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf46ee640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf46ee640 10 monclient(hunting): authenticate will time out at 3608.766202s 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf1c89640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaec0b1bf0 0x7fbaec0ab060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf1c89640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaec0b1bf0 0x7fbaec0ab060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42532/0 (socket says 172.21.15.106:42532) 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf1c89640 1 -- 172.21.15.106:0/1990104848 learned_addr learned my addr 172.21.15.106:0/1990104848 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf1c89640 10 monclient(hunting): get_auth_request con 0x7fbaec0b1bf0 auth_method 0 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf1c89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf1c89640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:19.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.499+0000 7fbaf1c89640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:19.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.500+0000 7fbaf1c89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:19.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.500+0000 7fbaf1c89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:19.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.500+0000 7fbaf1c89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:19.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.503+0000 7fbaf1c89640 10 monclient(hunting): handle_auth_done global_id 15162 payload 824 2024-04-04T21:39:19.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.503+0000 7fbaf1c89640 1 -- 172.21.15.106:0/1990104848 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbaec0afbd0 msgr2=0x7fbaec13e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:19.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.503+0000 7fbaf1c89640 1 --2- 172.21.15.106:0/1990104848 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbaec0afbd0 0x7fbaec13e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.503+0000 7fbaf1c89640 1 -- 172.21.15.106:0/1990104848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaec143100 msgr2=0x7fbaec0ae0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:19.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.503+0000 7fbaf1c89640 1 --2- 172.21.15.106:0/1990104848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaec143100 0x7fbaec0ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.503+0000 7fbaf1c89640 10 monclient: _finish_hunting 0 2024-04-04T21:39:19.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.503+0000 7fbaf1c89640 1 monclient: found mon.noname-a 2024-04-04T21:39:19.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.503+0000 7fbaf1c89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:19.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.503+0000 7fbaf1c89640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:39:19.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.503+0000 7fbaf1c89640 1 -- 172.21.15.106:0/1990104848 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaec0ad450 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.505+0000 7fbaf1c89640 1 --2- 172.21.15.106:0/1990104848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaec0b1bf0 0x7fbaec0ab060 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7fbadc02b8c0 tx=0x7fbadc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:19.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.505+0000 7fbae2ffd640 10 client.?.objecter ms_handle_connect 0x7fbaec0b1bf0 2024-04-04T21:39:19.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.505+0000 7fbae2ffd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:19.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbae2ffd640 1 -- 172.21.15.106:0/1990104848 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc032880 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbae2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:19.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbae2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:39:19.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbae2ffd640 10 monclient: dump: 2024-04-04T21:39:19.509 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:19.509 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:19.509 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.509 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.509 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:19.509 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:19.509 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:19.510 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:19.510 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:19.510 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:19.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbae2ffd640 10 monclient: _finish_auth 0 2024-04-04T21:39:19.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbae2ffd640 10 monclient: _check_auth_tickets 2024-04-04T21:39:19.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbae2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:19.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbae2ffd640 1 -- 172.21.15.106:0/1990104848 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc032a20 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbaf46ee640 5 monclient: authenticate success, global_id 15162 2024-04-04T21:39:19.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbae2ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:19.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbae2ffd640 1 -- 172.21.15.106:0/1990104848 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc032e10 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbaf46ee640 10 monclient: _renew_subs 2024-04-04T21:39:19.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbaf46ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:39:19.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbaf46ee640 1 -- 172.21.15.106:0/1990104848 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaec0ae5e0 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbaf0c87640 4 set_mon_vals no callback set 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbae2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbae2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbae2ffd640 10 monclient: dump: 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbaf46ee640 10 client.15162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbaf46ee640 10 monclient: _renew_subs 2024-04-04T21:39:19.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbaf46ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:39:19.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbaf46ee640 1 -- 172.21.15.106:0/1990104848 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaec1dbbe0 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.508+0000 7fbaf46ee640 1 librados: init done 2024-04-04T21:39:19.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.509+0000 7fbaf46ee640 10 client.15162.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:19.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.509+0000 7fbaf46ee640 15 client.15162.objecter _session_command_op_assign -1 1 2024-04-04T21:39:19.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.509+0000 7fbaf46ee640 20 client.15162.objecter _get_session osd=-1 returning homeless 2024-04-04T21:39:19.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.509+0000 7fbaf46ee640 10 client.15162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:19.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.509+0000 7fbaf46ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:39:19.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.509+0000 7fbaf46ee640 1 -- 172.21.15.106:0/1990104848 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbaec0685a0 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.529+0000 7fbae2ffd640 1 -- 172.21.15.106:0/1990104848 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fbadc0453f0 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.529+0000 7fbae2ffd640 1 --2- 172.21.15.106:0/1990104848 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fbad006ce90 0x7fbad006f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:19.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.529+0000 7fbaf248a640 1 --2- 172.21.15.106:0/1990104848 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fbad006ce90 0x7fbad006f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:19.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.529+0000 7fbaf248a640 10 monclient: get_auth_request con 0x7fbad006ce90 auth_method 0 2024-04-04T21:39:19.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.530+0000 7fbae2ffd640 10 client.15162.objecter ms_handle_connect 0x7fbad006ce90 2024-04-04T21:39:19.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.530+0000 7fbaf248a640 1 --2- 172.21.15.106:0/1990104848 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fbad006ce90 0x7fbad006f350 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fbae4009650 tx=0x7fbae4006ba0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:19.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 1 -- 172.21.15.106:0/1990104848 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7fbadc079010 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 10 client.15162.objecter ms_dispatch 0x7fbaec000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:19.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 3 client.15162.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:19.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 3 client.15162.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:19.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 10 client.15162.objecter checking command 1 2024-04-04T21:39:19.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 1 --2- 172.21.15.106:0/1990104848 >> [v2:172.21.15.203:6816/2758852830,v1:172.21.15.203:6817/2758852830] conn(0x7fbad0074ba0 0x7fbad0077040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:19.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 20 client.15162.objecter _get_session s=0x7fbad00744d0 osd=6 3 2024-04-04T21:39:19.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 20 client.15162.objecter put_session s=0x7fbad00744d0 osd=6 3 2024-04-04T21:39:19.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 15 client.15162.objecter _session_command_op_remove -1 1 2024-04-04T21:39:19.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 20 client.15162.objecter _get_session s=0x7fbad00744d0 osd=6 3 2024-04-04T21:39:19.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 20 client.15162.objecter get_session s=0x7fbad00744d0 osd=6 3 2024-04-04T21:39:19.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 15 client.15162.objecter _session_command_op_assign 6 1 2024-04-04T21:39:19.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 20 client.15162.objecter put_session s=0x7fbad00744d0 osd=6 4 2024-04-04T21:39:19.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 10 client.15162.objecter _send_command 1 2024-04-04T21:39:19.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 1 -- 172.21.15.106:0/1990104848 --> [v2:172.21.15.203:6816/2758852830,v1:172.21.15.203:6817/2758852830] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbad0077710 con 0x7fbad0074ba0 2024-04-04T21:39:19.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.533+0000 7fbae2ffd640 20 client.15162.objecter dump_active .. 0 homeless 2024-04-04T21:39:19.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.534+0000 7fbaf2c8b640 1 --2- 172.21.15.106:0/1990104848 >> [v2:172.21.15.203:6816/2758852830,v1:172.21.15.203:6817/2758852830] conn(0x7fbad0074ba0 0x7fbad0077040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:19.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.534+0000 7fbaf2c8b640 10 monclient: get_auth_request con 0x7fbad0074ba0 auth_method 0 2024-04-04T21:39:19.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.537+0000 7fbae2ffd640 10 client.15162.objecter ms_handle_connect 0x7fbad0074ba0 2024-04-04T21:39:19.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.537+0000 7fbaf2c8b640 1 --2- 172.21.15.106:0/1990104848 >> [v2:172.21.15.203:6816/2758852830,v1:172.21.15.203:6817/2758852830] conn(0x7fbad0074ba0 0x7fbad0077040 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:19.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.537+0000 7fbae2ffd640 1 -- 172.21.15.106:0/1990104848 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=1197) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbadc077210 con 0x7fbaec0b1bf0 2024-04-04T21:39:19.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.537+0000 7fbae2ffd640 10 monclient: handle_get_version_reply finishing 1 version 1197 2024-04-04T21:39:19.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.560+0000 7fbae2ffd640 1 -- 172.21.15.106:0/1990104848 <== osd.6 v2:172.21.15.203:6816/2758852830 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbad0077710 con 0x7fbad0074ba0 2024-04-04T21:39:19.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.560+0000 7fbae2ffd640 10 client.15162.objecter ms_dispatch 0x7fbaec000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:39:19.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.560+0000 7fbae2ffd640 10 client.15162.objecter _finish_command 1 = system:0 2024-04-04T21:39:19.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.560+0000 7fbae2ffd640 20 client.15162.objecter put_session s=0x7fbad00744d0 osd=6 4 2024-04-04T21:39:19.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.560+0000 7fbae2ffd640 15 client.15162.objecter _session_command_op_remove 6 1 2024-04-04T21:39:19.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.573+0000 7fbaf46ee640 10 client.15162.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-04T21:39:19.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.573+0000 7fbaf46ee640 15 client.15162.objecter _session_command_op_assign -1 2 2024-04-04T21:39:19.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.573+0000 7fbaf46ee640 20 client.15162.objecter _get_session s=0x7fbad00744d0 osd=6 3 2024-04-04T21:39:19.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.573+0000 7fbaf46ee640 20 client.15162.objecter put_session s=0x7fbad00744d0 osd=6 3 2024-04-04T21:39:19.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.573+0000 7fbaf46ee640 20 client.15162.objecter _get_session s=0x7fbad00744d0 osd=6 3 2024-04-04T21:39:19.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.573+0000 7fbaf46ee640 15 client.15162.objecter _session_command_op_remove -1 2 2024-04-04T21:39:19.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.573+0000 7fbaf46ee640 20 client.15162.objecter get_session s=0x7fbad00744d0 osd=6 3 2024-04-04T21:39:19.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.573+0000 7fbaf46ee640 15 client.15162.objecter _session_command_op_assign 6 2 2024-04-04T21:39:19.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.573+0000 7fbaf46ee640 20 client.15162.objecter put_session s=0x7fbad00744d0 osd=6 4 2024-04-04T21:39:19.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.573+0000 7fbaf46ee640 10 client.15162.objecter _send_command 2 2024-04-04T21:39:19.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.573+0000 7fbaf46ee640 1 -- 172.21.15.106:0/1990104848 --> [v2:172.21.15.203:6816/2758852830,v1:172.21.15.203:6817/2758852830] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbaec0b6580 con 0x7fbad0074ba0 2024-04-04T21:39:19.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.586+0000 7fbae2ffd640 1 -- 172.21.15.106:0/1990104848 <== osd.6 v2:172.21.15.203:6816/2758852830 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fbaec0b6580 con 0x7fbad0074ba0 2024-04-04T21:39:19.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.586+0000 7fbae2ffd640 10 client.15162.objecter ms_dispatch 0x7fbaec000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:39:19.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.586+0000 7fbae2ffd640 10 client.15162.objecter _finish_command 2 = system:0 2024-04-04T21:39:19.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.586+0000 7fbae2ffd640 20 client.15162.objecter put_session s=0x7fbad00744d0 osd=6 4 2024-04-04T21:39:19.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.586+0000 7fbae2ffd640 15 client.15162.objecter _session_command_op_remove 6 2 2024-04-04T21:39:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-04T21:39:19.588 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-04T21:39:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-04T21:39:19.589 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 10 librados: watch_flush enter 2024-04-04T21:39:19.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 10 librados: watch_flush exit 2024-04-04T21:39:19.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 10 client.15162.objecter close_session for osd.6 2024-04-04T21:39:19.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 1 -- 172.21.15.106:0/1990104848 >> [v2:172.21.15.203:6816/2758852830,v1:172.21.15.203:6817/2758852830] conn(0x7fbad0074ba0 msgr2=0x7fbad0077040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:19.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 1 --2- 172.21.15.106:0/1990104848 >> [v2:172.21.15.203:6816/2758852830,v1:172.21.15.203:6817/2758852830] conn(0x7fbad0074ba0 0x7fbad0077040 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 20 client.15162.objecter put_session s=0x7fbad00744d0 osd=6 1 2024-04-04T21:39:19.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 20 client.15162.objecter shutdown clearing up homeless session... 2024-04-04T21:39:19.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 10 client.15162.objecter successfully canceled tick 2024-04-04T21:39:19.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 1 -- 172.21.15.106:0/1990104848 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fbad006ce90 msgr2=0x7fbad006f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:19.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 1 --2- 172.21.15.106:0/1990104848 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fbad006ce90 0x7fbad006f350 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fbae4009650 tx=0x7fbae4006ba0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 10 monclient: shutdown 2024-04-04T21:39:19.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:19.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 1 -- 172.21.15.106:0/1990104848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaec0b1bf0 msgr2=0x7fbaec0ab060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:19.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 1 --2- 172.21.15.106:0/1990104848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaec0b1bf0 0x7fbaec0ab060 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7fbadc02b8c0 tx=0x7fbadc0046c0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf2c8b640 1 -- 172.21.15.106:0/1990104848 reap_dead start 2024-04-04T21:39:19.591 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 1 -- 172.21.15.106:0/1990104848 shutdown_connections 2024-04-04T21:39:19.591 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 1 -- 172.21.15.106:0/1990104848 >> 172.21.15.106:0/1990104848 conn(0x7fbaec06d3a0 msgr2=0x7fbaec0b8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:19.591 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 1 -- 172.21.15.106:0/1990104848 shutdown_connections 2024-04-04T21:39:19.591 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 1 -- 172.21.15.106:0/1990104848 wait complete. 2024-04-04T21:39:19.591 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.587+0000 7fbaf46ee640 1 librados: shutdown 2024-04-04T21:39:19.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-04T21:39:19.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.700+0000 7f04f08ee640 10 monclient: get_monmap_and_config 2024-04-04T21:39:19.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.700+0000 7f04f08ee640 10 monclient: build_initial_monmap 2024-04-04T21:39:19.701 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.700+0000 7f04f08ee640 10 monclient: monmap: 2024-04-04T21:39:19.701 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:19.701 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:19.701 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:19.701947+0000 2024-04-04T21:39:19.701 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:19.701947+0000 2024-04-04T21:39:19.701 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:19.701 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:19.701 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:19.701 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:19.701 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:19.702 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:19.702 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:19.702 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:19.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.701+0000 7f04f08ee640 1 Processor -- start 2024-04-04T21:39:19.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.701+0000 7f04f08ee640 1 -- start start 2024-04-04T21:39:19.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.701+0000 7f04f08ee640 10 monclient: init 2024-04-04T21:39:19.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04f08ee640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:19.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04f08ee640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-04-04T21:39:19.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04f08ee640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f04ec0a9a50 0x7f04ec0a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:19.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04f08ee640 10 monclient(hunting): picked mon.noname-c con 0x7f04ec0a9a50 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:19.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04f08ee640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f04ec13f6a0 0x7f04ec13fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:19.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04f08ee640 10 monclient(hunting): picked mon.noname-a con 0x7f04ec13f6a0 addr v2:172.21.15.106:3300/0 2024-04-04T21:39:19.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04f08ee640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04ec142880 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:19.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04f08ee640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ec0ab750 con 0x7f04ec142880 2024-04-04T21:39:19.704 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04f08ee640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04f08ee640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ec0aa370 con 0x7f04ec13f6a0 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04f08ee640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04f08ee640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ec143400 con 0x7f04ec0a9a50 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04f08ee640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04f08ee640 10 monclient(hunting): authenticate will time out at 3608.969564s 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.702+0000 7f04e9d9b640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f04ec13f6a0 0x7f04ec13fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04ea59c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f04ec0a9a50 0x7f04ec0a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04e9d9b640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f04ec13f6a0 0x7f04ec13fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42534/0 (socket says 172.21.15.106:42534) 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04e9d9b640 1 -- 172.21.15.106:0/3626367716 learned_addr learned my addr 172.21.15.106:0/3626367716 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04e9d9b640 10 monclient(hunting): get_auth_request con 0x7f04ec13f6a0 auth_method 0 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04e9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04ea59c640 10 monclient(hunting): get_auth_request con 0x7f04ec0a9a50 auth_method 0 2024-04-04T21:39:19.705 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:19.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04ea59c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:19.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:19.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04e9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:19.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04e9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:19.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04e9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:19.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:19.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:19.706 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.703+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:19.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.705+0000 7f04ea59c640 10 monclient(hunting): handle_auth_done global_id 22123 payload 274 2024-04-04T21:39:19.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.705+0000 7f04ea59c640 1 -- 172.21.15.106:0/3626367716 >> v2:172.21.15.106:3300/0 conn(0x7f04ec13f6a0 msgr2=0x7f04ec13fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:19.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.705+0000 7f04ea59c640 1 --2- 172.21.15.106:0/3626367716 >> v2:172.21.15.106:3300/0 conn(0x7f04ec13f6a0 0x7f04ec13fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.705+0000 7f04ea59c640 1 -- 172.21.15.106:0/3626367716 >> v1:172.21.15.106:6789/0 conn(0x7f04ec142880 legacy=0x7f04ec142cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:19.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.705+0000 7f04ea59c640 10 monclient: _finish_hunting 0 2024-04-04T21:39:19.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.705+0000 7f04ea59c640 1 monclient: found mon.noname-c 2024-04-04T21:39:19.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.705+0000 7f04ea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:19.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.705+0000 7f04ea59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:39:19.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.705+0000 7f04ea59c640 1 -- 172.21.15.106:0/3626367716 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec140120 con 0x7f04ec0a9a50 2024-04-04T21:39:19.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.705+0000 7f04e9d9b640 1 --2- 172.21.15.106:0/3626367716 >> v2:172.21.15.106:3300/0 conn(0x7f04ec13f6a0 0x7f04ec13fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-04T21:39:19.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.707+0000 7f04ea59c640 1 --2- 172.21.15.106:0/3626367716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ec0a9a50 0x7f04ec0a9e30 secure :-1 s=READY pgs=5553 cs=0 l=1 rev1=1 crypto rx=0x7f04ec06c340 tx=0x7f04d402b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd74f9199f4b1f0b server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:19.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04e959a640 1 -- 172.21.15.106:0/3626367716 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4038070 con 0x7f04ec0a9a50 2024-04-04T21:39:19.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:19.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04e959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:39:19.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04e959a640 10 monclient: dump: 2024-04-04T21:39:19.711 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:19.711 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:19.711 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.711 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.711 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04e959a640 10 monclient: _finish_auth 0 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04e959a640 10 monclient: _check_auth_tickets 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04e959a640 1 -- 172.21.15.106:0/3626367716 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d4031710 con 0x7f04ec0a9a50 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04e959a640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04e959a640 1 -- 172.21.15.106:0/3626367716 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4031ae0 con 0x7f04ec0a9a50 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04f08ee640 5 monclient: authenticate success, global_id 22123 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04f08ee640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04f08ee640 4 set_mon_vals no callback set 2024-04-04T21:39:19.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04e959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:19.713 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04e959a640 10 monclient: dump: 2024-04-04T21:39:19.713 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:19.713 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:19.713 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.713 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.713 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:19.713 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:19.713 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:19.713 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:19.713 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:19.713 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:19.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04f08ee640 10 monclient: shutdown 2024-04-04T21:39:19.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04f08ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:19.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04f08ee640 1 -- 172.21.15.106:0/3626367716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ec0a9a50 msgr2=0x7f04ec0a9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:19.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04f08ee640 1 --2- 172.21.15.106:0/3626367716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ec0a9a50 0x7f04ec0a9e30 secure :-1 s=READY pgs=5553 cs=0 l=1 rev1=1 crypto rx=0x7f04ec06c340 tx=0x7f04d402b7b0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04f08ee640 1 -- 172.21.15.106:0/3626367716 shutdown_connections 2024-04-04T21:39:19.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04f08ee640 1 --2- 172.21.15.106:0/3626367716 >> v2:172.21.15.106:3300/0 conn(0x7f04ec13f6a0 0x7f04ec13fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04f08ee640 1 --2- 172.21.15.106:0/3626367716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ec0a9a50 0x7f04ec0a9e30 unknown :-1 s=CLOSED pgs=5553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04f08ee640 1 -- 172.21.15.106:0/3626367716 >> 172.21.15.106:0/3626367716 conn(0x7f04ec0b8030 msgr2=0x7f04ec0b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04f08ee640 1 -- 172.21.15.106:0/3626367716 shutdown_connections 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.711+0000 7f04f08ee640 1 -- 172.21.15.106:0/3626367716 wait complete. 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.712+0000 7f04f08ee640 10 monclient: build_initial_monmap 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.712+0000 7f04f08ee640 10 monclient: monmap: 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.712+0000 7f04f08ee640 1 librados: starting msgr at 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.712+0000 7f04f08ee640 1 librados: starting objecter 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 1 Processor -- start 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 1 -- start start 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 1 librados: setting wanted keys 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 1 librados: calling monclient init 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 10 monclient: init 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04ec0a9a50 0x7f04ec13c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 10 monclient(hunting): picked mon.noname-a con 0x7f04ec0a9a50 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ec13f6a0 0x7f04ec146020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 10 monclient(hunting): picked mon.noname-c con 0x7f04ec13f6a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ec142880 0x7f04ec139590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 10 monclient(hunting): picked mon.noname-b con 0x7f04ec142880 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:19.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec14d070 con 0x7f04ec0a9a50 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec14cef0 con 0x7f04ec142880 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec14d1f0 con 0x7f04ec13f6a0 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.713+0000 7f04f08ee640 10 monclient(hunting): authenticate will time out at 3608.980925s 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04ea59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04ec0a9a50 0x7f04ec13c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04e9d9b640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ec13f6a0 0x7f04ec146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04ead9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ec142880 0x7f04ec139590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04ea59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04ec0a9a50 0x7f04ec13c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42550/0 (socket says 172.21.15.106:42550) 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04ea59c640 1 -- 172.21.15.106:0/2531949339 learned_addr learned my addr 172.21.15.106:0/2531949339 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04ea59c640 10 monclient(hunting): get_auth_request con 0x7f04ec0a9a50 auth_method 0 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:19.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04ea59c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04e9d9b640 10 monclient(hunting): get_auth_request con 0x7f04ec13f6a0 auth_method 0 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04e9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04ead9d640 10 monclient(hunting): get_auth_request con 0x7f04ec142880 auth_method 0 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04ead9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.714+0000 7f04ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.715+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.715+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.715+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.715+0000 7f04ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.715+0000 7f04ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:19.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.715+0000 7f04ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:19.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.715+0000 7f04e9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:19.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.715+0000 7f04e9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:19.722 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.715+0000 7f04e9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:19.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.717+0000 7f04e9d9b640 10 monclient(hunting): handle_auth_done global_id 17696 payload 824 2024-04-04T21:39:19.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.717+0000 7f04e9d9b640 1 -- 172.21.15.106:0/2531949339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ec142880 msgr2=0x7f04ec139590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:19.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.717+0000 7f04e9d9b640 1 --2- 172.21.15.106:0/2531949339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ec142880 0x7f04ec139590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.718+0000 7f04e9d9b640 1 -- 172.21.15.106:0/2531949339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04ec0a9a50 msgr2=0x7f04ec13c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:19.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.718+0000 7f04e9d9b640 1 --2- 172.21.15.106:0/2531949339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04ec0a9a50 0x7f04ec13c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:19.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.718+0000 7f04e9d9b640 10 monclient: _finish_hunting 0 2024-04-04T21:39:19.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.718+0000 7f04e9d9b640 1 monclient: found mon.noname-c 2024-04-04T21:39:19.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.718+0000 7f04e9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:19.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.718+0000 7f04e9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.106:3301/0 2024-04-04T21:39:19.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.718+0000 7f04e9d9b640 1 -- 172.21.15.106:0/2531949339 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec145340 con 0x7f04ec13f6a0 2024-04-04T21:39:19.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.720+0000 7f04e9d9b640 1 --2- 172.21.15.106:0/2531949339 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ec13f6a0 0x7f04ec146020 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00e900 tx=0x7f04dc00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:19.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.720+0000 7f04e2ffd640 10 client.?.objecter ms_handle_connect 0x7f04ec13f6a0 2024-04-04T21:39:19.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.720+0000 7f04e2ffd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04e2ffd640 1 -- 172.21.15.106:0/2531949339 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc00ca00 con 0x7f04ec13f6a0 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04e2ffd640 10 monclient: dump: 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:19.725 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04e2ffd640 10 monclient: _finish_auth 0 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04e2ffd640 10 monclient: _check_auth_tickets 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04f08ee640 5 monclient: authenticate success, global_id 17696 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04f08ee640 10 monclient: _renew_subs 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04f08ee640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04f08ee640 1 -- 172.21.15.106:0/2531949339 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ec146560 con 0x7f04ec13f6a0 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04f08ee640 10 client.17696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04f08ee640 10 monclient: _renew_subs 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04f08ee640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04f08ee640 1 -- 172.21.15.106:0/2531949339 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ec1467d0 con 0x7f04ec13f6a0 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.722+0000 7f04f08ee640 1 librados: init done 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.723+0000 7f04f08ee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.723+0000 7f04f08ee640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.723+0000 7f04f08ee640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.106:3301/0 2024-04-04T21:39:19.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.723+0000 7f04f08ee640 1 -- 172.21.15.106:0/2531949339 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ec04f050 con 0x7f04ec13f6a0 2024-04-04T21:39:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.971+0000 7f04e2ffd640 1 -- 172.21.15.106:0/2531949339 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04dc0093b0 con 0x7f04ec13f6a0 2024-04-04T21:39:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.971+0000 7f04e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.971+0000 7f04e2ffd640 1 -- 172.21.15.106:0/2531949339 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc020060 con 0x7f04ec13f6a0 2024-04-04T21:39:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.971+0000 7f04e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.971+0000 7f04e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:39:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.971+0000 7f04e2ffd640 10 monclient: dump: 2024-04-04T21:39:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:19.972 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:19.973 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:19.973 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:19.973 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:19.973 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:19.973 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.971+0000 7f04e3fff640 4 set_mon_vals no callback set 2024-04-04T21:39:19.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.972+0000 7f04e2ffd640 1 -- 172.21.15.106:0/2531949339 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f04dc004030 con 0x7f04ec13f6a0 2024-04-04T21:39:19.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.972+0000 7f04e2ffd640 1 --2- 172.21.15.106:0/2531949339 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f04cc06ce60 0x7f04cc06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:19.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.973+0000 7f04ea59c640 1 --2- 172.21.15.106:0/2531949339 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f04cc06ce60 0x7f04cc06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:19.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.973+0000 7f04ea59c640 10 monclient: get_auth_request con 0x7f04cc06ce60 auth_method 0 2024-04-04T21:39:19.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.973+0000 7f04ea59c640 1 --2- 172.21.15.106:0/2531949339 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f04cc06ce60 0x7f04cc06f320 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f04d4009560 tx=0x7f04d4008030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:19.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.973+0000 7f04e2ffd640 10 client.17696.objecter ms_handle_connect 0x7f04cc06ce60 2024-04-04T21:39:19.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.985+0000 7f04e2ffd640 1 -- 172.21.15.106:0/2531949339 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f04dc057210 con 0x7f04ec13f6a0 2024-04-04T21:39:19.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.985+0000 7f04e2ffd640 10 client.17696.objecter ms_dispatch 0x7f04ec000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:19.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.985+0000 7f04e2ffd640 3 client.17696.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:19.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.985+0000 7f04e2ffd640 3 client.17696.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:19.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.985+0000 7f04e2ffd640 20 client.17696.objecter dump_active .. 0 homeless 2024-04-04T21:39:19.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.985+0000 7f04e2ffd640 1 -- 172.21.15.106:0/2531949339 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f04dc017020 con 0x7f04ec13f6a0 2024-04-04T21:39:19.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.985+0000 7f04e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:19.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:19.985+0000 7f04e2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:20.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.159+0000 7f04f08ee640 1 -- 172.21.15.106:0/2531949339 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f04ec0b7220 con 0x7f04cc06ce60 2024-04-04T21:39:20.160 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.160+0000 7f04e2ffd640 1 -- 172.21.15.106:0/2531949339 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f04ec0b7220 con 0x7f04cc06ce60 2024-04-04T21:39:20.160 INFO:teuthology.orchestra.run.smithi106.stderr:instructed osd(s) 6 to deep-scrub 2024-04-04T21:39:20.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.161+0000 7f04f08ee640 10 librados: watch_flush enter 2024-04-04T21:39:20.163 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.161+0000 7f04f08ee640 10 librados: watch_flush exit 2024-04-04T21:39:20.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.161+0000 7f04f08ee640 20 client.17696.objecter shutdown clearing up homeless session... 2024-04-04T21:39:20.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.161+0000 7f04f08ee640 10 client.17696.objecter successfully canceled tick 2024-04-04T21:39:20.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 1 -- 172.21.15.106:0/2531949339 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f04cc06ce60 msgr2=0x7f04cc06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:20.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 1 --2- 172.21.15.106:0/2531949339 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f04cc06ce60 0x7f04cc06f320 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f04d4009560 tx=0x7f04d4008030 comp rx=0 tx=0).stop 2024-04-04T21:39:20.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 10 monclient: shutdown 2024-04-04T21:39:20.164 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:20.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 1 -- 172.21.15.106:0/2531949339 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ec13f6a0 msgr2=0x7f04ec146020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:20.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 1 --2- 172.21.15.106:0/2531949339 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ec13f6a0 0x7f04ec146020 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00e900 tx=0x7f04dc00edc0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 1 -- 172.21.15.106:0/2531949339 shutdown_connections 2024-04-04T21:39:20.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 1 --2- 172.21.15.106:0/2531949339 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f04cc06ce60 0x7f04cc06f320 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 1 --2- 172.21.15.106:0/2531949339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ec142880 0x7f04ec139590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.165 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 1 --2- 172.21.15.106:0/2531949339 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ec13f6a0 0x7f04ec146020 unknown :-1 s=CLOSED pgs=3274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 1 --2- 172.21.15.106:0/2531949339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04ec0a9a50 0x7f04ec13c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 1 -- 172.21.15.106:0/2531949339 >> 172.21.15.106:0/2531949339 conn(0x7f04ec0b8030 msgr2=0x7f04ec0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:20.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 1 -- 172.21.15.106:0/2531949339 shutdown_connections 2024-04-04T21:39:20.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 1 -- 172.21.15.106:0/2531949339 wait complete. 2024-04-04T21:39:20.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.162+0000 7f04f08ee640 1 librados: shutdown 2024-04-04T21:39:20.174 INFO:tasks.ceph:Scrubbing osd.7 2024-04-04T21:39:20.174 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-04T21:39:20.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.278+0000 7f308ac14640 10 monclient: get_monmap_and_config 2024-04-04T21:39:20.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.278+0000 7f308ac14640 10 monclient: build_initial_monmap 2024-04-04T21:39:20.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.278+0000 7f308ac14640 10 monclient: monmap: 2024-04-04T21:39:20.279 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:20.279 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:20.279 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:20.279129+0000 2024-04-04T21:39:20.279 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:20.279129+0000 2024-04-04T21:39:20.279 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:20.279 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:20.279 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:20.279 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:20.280 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:20.280 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:20.280 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:20.280 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:20.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.278+0000 7f308ac14640 1 Processor -- start 2024-04-04T21:39:20.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.278+0000 7f308ac14640 1 -- start start 2024-04-04T21:39:20.280 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.278+0000 7f308ac14640 10 monclient: init 2024-04-04T21:39:20.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.279+0000 7f308ac14640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:20.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.279+0000 7f308ac14640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-04-04T21:39:20.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.279+0000 7f308ac14640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f30840b5360 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:20.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.279+0000 7f308ac14640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308413fc70 0x7f3084145b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:20.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.279+0000 7f308ac14640 10 monclient(hunting): picked mon.noname-b con 0x7f308413fc70 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:20.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.279+0000 7f308ac14640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3084146090 0x7f308414cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:20.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.279+0000 7f308ac14640 10 monclient(hunting): picked mon.noname-a con 0x7f3084146090 addr v2:172.21.15.106:3300/0 2024-04-04T21:39:20.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.279+0000 7f308ac14640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30840ab3e0 con 0x7f30840b5360 2024-04-04T21:39:20.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.279+0000 7f308ac14640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:20.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.279+0000 7f308ac14640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30840b5740 con 0x7f3084146090 2024-04-04T21:39:20.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.279+0000 7f308ac14640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:20.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.279+0000 7f308ac14640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30840a9e70 con 0x7f308413fc70 2024-04-04T21:39:20.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.279+0000 7f308ac14640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:20.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.279+0000 7f308ac14640 10 monclient(hunting): authenticate will time out at 3609.546632s 2024-04-04T21:39:20.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.280+0000 7f3083fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308413fc70 0x7f3084145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:20.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.280+0000 7f30891b1640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3084146090 0x7f308414cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:20.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.280+0000 7f30891b1640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3084146090 0x7f308414cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42556/0 (socket says 172.21.15.106:42556) 2024-04-04T21:39:20.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.280+0000 7f30891b1640 1 -- 172.21.15.106:0/3330238613 learned_addr learned my addr 172.21.15.106:0/3330238613 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:20.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.280+0000 7f30891b1640 10 monclient(hunting): get_auth_request con 0x7f3084146090 auth_method 0 2024-04-04T21:39:20.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.280+0000 7f30891b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:20.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.280+0000 7f30891b1640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:20.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.280+0000 7f30891b1640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:20.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.280+0000 7f3083fff640 10 monclient(hunting): get_auth_request con 0x7f308413fc70 auth_method 0 2024-04-04T21:39:20.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.280+0000 7f3083fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:20.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.280+0000 7f3083fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:20.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.280+0000 7f3083fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:20.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.280+0000 7f30891b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:20.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.280+0000 7f30891b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:20.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.281+0000 7f30891b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:20.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.281+0000 7f3083fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:20.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.281+0000 7f3083fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:20.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.281+0000 7f3083fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:20.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.282+0000 7f30891b1640 10 monclient(hunting): handle_auth_done global_id 15171 payload 274 2024-04-04T21:39:20.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.282+0000 7f30891b1640 1 -- 172.21.15.106:0/3330238613 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308413fc70 msgr2=0x7f3084145b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:20.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.282+0000 7f30891b1640 1 --2- 172.21.15.106:0/3330238613 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308413fc70 0x7f3084145b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.282+0000 7f30891b1640 1 -- 172.21.15.106:0/3330238613 >> v1:172.21.15.106:6789/0 conn(0x7f30840b5360 legacy=0x7f308413f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:20.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.282+0000 7f30891b1640 10 monclient: _finish_hunting 0 2024-04-04T21:39:20.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.282+0000 7f30891b1640 1 monclient: found mon.noname-a 2024-04-04T21:39:20.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.282+0000 7f30891b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:20.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.282+0000 7f30891b1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:39:20.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.282+0000 7f30891b1640 1 -- 172.21.15.106:0/3330238613 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30840ab6c0 con 0x7f3084146090 2024-04-04T21:39:20.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.282+0000 7f3083fff640 1 --2- 172.21.15.106:0/3330238613 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308413fc70 0x7f3084145b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-04T21:39:20.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.284+0000 7f30891b1640 1 --2- 172.21.15.106:0/3330238613 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3084146090 0x7f308414cee0 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 crypto rx=0x7f307c009e50 tx=0x7f307c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e990ec1b2cb9d285 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:20.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f30837fe640 1 -- 172.21.15.106:0/3330238613 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c02c950 con 0x7f3084146090 2024-04-04T21:39:20.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f30837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:20.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f30837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:39:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f30837fe640 10 monclient: dump: 2024-04-04T21:39:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:20.291 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:20.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f30837fe640 10 monclient: _finish_auth 0 2024-04-04T21:39:20.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f30837fe640 10 monclient: _check_auth_tickets 2024-04-04T21:39:20.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f30837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:20.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f30837fe640 1 -- 172.21.15.106:0/3330238613 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307c02caf0 con 0x7f3084146090 2024-04-04T21:39:20.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f30837fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:20.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f30837fe640 1 -- 172.21.15.106:0/3330238613 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c0343f0 con 0x7f3084146090 2024-04-04T21:39:20.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f308ac14640 5 monclient: authenticate success, global_id 15171 2024-04-04T21:39:20.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f308ac14640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:20.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f308ac14640 4 set_mon_vals no callback set 2024-04-04T21:39:20.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f30837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:20.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f30837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:39:20.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.288+0000 7f30837fe640 10 monclient: dump: 2024-04-04T21:39:20.293 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:20.293 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:20.293 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.293 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.293 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:20.293 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:20.294 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:20.294 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:20.294 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:20.294 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:20.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.289+0000 7f308ac14640 10 monclient: shutdown 2024-04-04T21:39:20.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.289+0000 7f308ac14640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:20.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.289+0000 7f308ac14640 1 -- 172.21.15.106:0/3330238613 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3084146090 msgr2=0x7f308414cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:20.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.289+0000 7f308ac14640 1 --2- 172.21.15.106:0/3330238613 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3084146090 0x7f308414cee0 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 crypto rx=0x7f307c009e50 tx=0x7f307c02bdb0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.289+0000 7f308ac14640 1 -- 172.21.15.106:0/3330238613 shutdown_connections 2024-04-04T21:39:20.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.289+0000 7f308ac14640 1 --2- 172.21.15.106:0/3330238613 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3084146090 0x7f308414cee0 unknown :-1 s=CLOSED pgs=3169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.289+0000 7f308ac14640 1 --2- 172.21.15.106:0/3330238613 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308413fc70 0x7f3084145b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.289+0000 7f308ac14640 1 -- 172.21.15.106:0/3330238613 >> 172.21.15.106:0/3330238613 conn(0x7f3084066e70 msgr2=0x7f3084067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:20.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.289+0000 7f308ac14640 1 -- 172.21.15.106:0/3330238613 shutdown_connections 2024-04-04T21:39:20.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.289+0000 7f308ac14640 1 -- 172.21.15.106:0/3330238613 wait complete. 2024-04-04T21:39:20.302 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.290+0000 7f308ac14640 10 monclient: build_initial_monmap 2024-04-04T21:39:20.302 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.290+0000 7f308ac14640 10 monclient: monmap: 2024-04-04T21:39:20.302 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:20.302 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:20.302 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:20.302 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:20.302 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:20.302 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:20.302 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:20.303 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:20.303 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:20.303 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:20.303 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.290+0000 7f308ac14640 1 librados: starting msgr at 2024-04-04T21:39:20.303 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.290+0000 7f308ac14640 1 librados: starting objecter 2024-04-04T21:39:20.303 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.290+0000 7f308ac14640 1 Processor -- start 2024-04-04T21:39:20.303 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.290+0000 7f308ac14640 1 -- start start 2024-04-04T21:39:20.303 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.290+0000 7f308ac14640 1 librados: setting wanted keys 2024-04-04T21:39:20.303 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.290+0000 7f308ac14640 1 librados: calling monclient init 2024-04-04T21:39:20.303 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.290+0000 7f308ac14640 10 monclient: init 2024-04-04T21:39:20.303 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:20.303 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:39:20.303 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30840b5360 0x7f30841cf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:20.303 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 10 monclient(hunting): picked mon.noname-a con 0x7f30840b5360 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:20.304 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308413fc70 0x7f30841ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:20.304 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 10 monclient(hunting): picked mon.noname-c con 0x7f308413fc70 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:20.304 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3084146090 0x7f30841cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:20.304 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 10 monclient(hunting): picked mon.noname-b con 0x7f3084146090 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:20.304 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:20.304 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3084140110 con 0x7f30840b5360 2024-04-04T21:39:20.304 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:20.304 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3084146520 con 0x7f3084146090 2024-04-04T21:39:20.304 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:20.304 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30840b2650 con 0x7f308413fc70 2024-04-04T21:39:20.304 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.291+0000 7f308ac14640 10 monclient(hunting): authenticate will time out at 3609.558769s 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f30889b0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30840b5360 0x7f30841cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f3083fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308413fc70 0x7f30841ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f30891b1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3084146090 0x7f30841cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f30889b0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30840b5360 0x7f30841cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42570/0 (socket says 172.21.15.106:42570) 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f30889b0640 1 -- 172.21.15.106:0/2399518757 learned_addr learned my addr 172.21.15.106:0/2399518757 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f30889b0640 10 monclient(hunting): get_auth_request con 0x7f30840b5360 auth_method 0 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f30889b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f30889b0640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f30889b0640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f3083fff640 10 monclient(hunting): get_auth_request con 0x7f308413fc70 auth_method 0 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f3083fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f3083fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f3083fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:20.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f30891b1640 10 monclient(hunting): get_auth_request con 0x7f3084146090 auth_method 0 2024-04-04T21:39:20.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f30891b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:20.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f30891b1640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:20.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.292+0000 7f30891b1640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:20.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.293+0000 7f30889b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:20.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.293+0000 7f30889b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:20.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.293+0000 7f30889b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:20.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.293+0000 7f3083fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:20.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.293+0000 7f3083fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:20.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.293+0000 7f3083fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:20.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.293+0000 7f30891b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:20.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.293+0000 7f30891b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:20.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.293+0000 7f30891b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:20.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.295+0000 7f30889b0640 10 monclient(hunting): handle_auth_done global_id 15174 payload 824 2024-04-04T21:39:20.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.295+0000 7f30889b0640 1 -- 172.21.15.106:0/2399518757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308413fc70 msgr2=0x7f30841ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:20.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.295+0000 7f30889b0640 1 --2- 172.21.15.106:0/2399518757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308413fc70 0x7f30841ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.295+0000 7f30889b0640 1 -- 172.21.15.106:0/2399518757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3084146090 msgr2=0x7f30841cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:20.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.295+0000 7f30889b0640 1 --2- 172.21.15.106:0/2399518757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3084146090 0x7f30841cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.295+0000 7f30889b0640 10 monclient: _finish_hunting 0 2024-04-04T21:39:20.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.295+0000 7f30889b0640 1 monclient: found mon.noname-a 2024-04-04T21:39:20.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.295+0000 7f30889b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:20.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.295+0000 7f30889b0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:39:20.308 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.295+0000 7f30889b0640 1 -- 172.21.15.106:0/2399518757 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30841ce9d0 con 0x7f30840b5360 2024-04-04T21:39:20.308 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.298+0000 7f30889b0640 1 --2- 172.21.15.106:0/2399518757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30840b5360 0x7f30841cf440 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 crypto rx=0x7f307400c440 tx=0x7f307400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:20.308 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.298+0000 7f30817fa640 10 client.?.objecter ms_handle_connect 0x7f30840b5360 2024-04-04T21:39:20.308 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.298+0000 7f30817fa640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:20.308 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f30817fa640 1 -- 172.21.15.106:0/2399518757 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3074017070 con 0x7f30840b5360 2024-04-04T21:39:20.308 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f30817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:20.308 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f30817fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:39:20.308 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f30817fa640 10 monclient: dump: 2024-04-04T21:39:20.308 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:20.309 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:20.309 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.309 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.309 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:20.309 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:20.309 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:20.309 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:20.309 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:20.309 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:20.309 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f30817fa640 10 monclient: _finish_auth 0 2024-04-04T21:39:20.309 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f30817fa640 10 monclient: _check_auth_tickets 2024-04-04T21:39:20.309 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f30817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:20.309 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f30817fa640 1 -- 172.21.15.106:0/2399518757 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30740035f0 con 0x7f30840b5360 2024-04-04T21:39:20.309 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f30817fa640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:20.310 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f30817fa640 1 -- 172.21.15.106:0/2399518757 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30740039e0 con 0x7f30840b5360 2024-04-04T21:39:20.310 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f30817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:20.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f30817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:39:20.311 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f30817fa640 10 monclient: dump: 2024-04-04T21:39:20.311 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:20.311 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:20.311 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.311 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.312 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:20.312 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:20.312 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:20.312 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:20.312 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:20.312 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:20.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f308ac14640 5 monclient: authenticate success, global_id 15174 2024-04-04T21:39:20.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f308ac14640 10 monclient: _renew_subs 2024-04-04T21:39:20.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f308ac14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:39:20.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f308ac14640 1 -- 172.21.15.106:0/2399518757 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30841cecb0 con 0x7f30840b5360 2024-04-04T21:39:20.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f3082ffd640 4 set_mon_vals no callback set 2024-04-04T21:39:20.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f308ac14640 10 client.15174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:20.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f308ac14640 10 monclient: _renew_subs 2024-04-04T21:39:20.313 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f308ac14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:39:20.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f308ac14640 1 -- 172.21.15.106:0/2399518757 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3084139eb0 con 0x7f30840b5360 2024-04-04T21:39:20.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.301+0000 7f308ac14640 1 librados: init done 2024-04-04T21:39:20.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.302+0000 7f308ac14640 10 client.15174.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:20.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.302+0000 7f308ac14640 15 client.15174.objecter _session_command_op_assign -1 1 2024-04-04T21:39:20.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.302+0000 7f308ac14640 20 client.15174.objecter _get_session osd=-1 returning homeless 2024-04-04T21:39:20.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.302+0000 7f308ac14640 10 client.15174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:20.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.302+0000 7f308ac14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.106:3300/0 2024-04-04T21:39:20.315 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.302+0000 7f308ac14640 1 -- 172.21.15.106:0/2399518757 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3084069c40 con 0x7f30840b5360 2024-04-04T21:39:20.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.322+0000 7f30817fa640 1 -- 172.21.15.106:0/2399518757 <== mon.0 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f307401a070 con 0x7f30840b5360 2024-04-04T21:39:20.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.322+0000 7f30817fa640 1 --2- 172.21.15.106:0/2399518757 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f307006ceb0 0x7f307006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:20.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.323+0000 7f3083fff640 1 --2- 172.21.15.106:0/2399518757 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f307006ceb0 0x7f307006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:20.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.323+0000 7f3083fff640 10 monclient: get_auth_request con 0x7f307006ceb0 auth_method 0 2024-04-04T21:39:20.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.323+0000 7f30817fa640 10 client.15174.objecter ms_handle_connect 0x7f307006ceb0 2024-04-04T21:39:20.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.323+0000 7f3083fff640 1 --2- 172.21.15.106:0/2399518757 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f307006ceb0 0x7f307006f370 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3078009560 tx=0x7f3078009190 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:20.330 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 1 -- 172.21.15.106:0/2399518757 <== mon.0 v2:172.21.15.106:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f3074056230 con 0x7f30840b5360 2024-04-04T21:39:20.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 10 client.15174.objecter ms_dispatch 0x7f3084000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:20.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 3 client.15174.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:20.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 3 client.15174.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:20.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 10 client.15174.objecter checking command 1 2024-04-04T21:39:20.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 1 --2- 172.21.15.106:0/2399518757 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x7f3070074bc0 0x7f3070077060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:20.331 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 20 client.15174.objecter _get_session s=0x7f30700744f0 osd=7 3 2024-04-04T21:39:20.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 20 client.15174.objecter put_session s=0x7f30700744f0 osd=7 3 2024-04-04T21:39:20.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 15 client.15174.objecter _session_command_op_remove -1 1 2024-04-04T21:39:20.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 20 client.15174.objecter _get_session s=0x7f30700744f0 osd=7 3 2024-04-04T21:39:20.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 20 client.15174.objecter get_session s=0x7f30700744f0 osd=7 3 2024-04-04T21:39:20.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 15 client.15174.objecter _session_command_op_assign 7 1 2024-04-04T21:39:20.332 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 20 client.15174.objecter put_session s=0x7f30700744f0 osd=7 4 2024-04-04T21:39:20.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 10 client.15174.objecter _send_command 1 2024-04-04T21:39:20.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.327+0000 7f30817fa640 1 -- 172.21.15.106:0/2399518757 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3070077730 con 0x7f3070074bc0 2024-04-04T21:39:20.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.328+0000 7f30817fa640 20 client.15174.objecter dump_active .. 0 homeless 2024-04-04T21:39:20.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.328+0000 7f30817fa640 1 -- 172.21.15.106:0/2399518757 <== mon.0 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=1197) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3074054430 con 0x7f30840b5360 2024-04-04T21:39:20.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.328+0000 7f30817fa640 10 monclient: handle_get_version_reply finishing 1 version 1197 2024-04-04T21:39:20.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.328+0000 7f30891b1640 1 --2- 172.21.15.106:0/2399518757 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x7f3070074bc0 0x7f3070077060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:20.333 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.329+0000 7f30891b1640 10 monclient: get_auth_request con 0x7f3070074bc0 auth_method 0 2024-04-04T21:39:20.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.331+0000 7f30891b1640 1 --2- 172.21.15.106:0/2399518757 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x7f3070074bc0 0x7f3070077060 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:20.334 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.331+0000 7f30817fa640 10 client.15174.objecter ms_handle_connect 0x7f3070074bc0 2024-04-04T21:39:20.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.356+0000 7f30817fa640 1 -- 172.21.15.106:0/2399518757 <== osd.7 v2:172.21.15.203:6808/4104640840 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3070077730 con 0x7f3070074bc0 2024-04-04T21:39:20.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.356+0000 7f30817fa640 10 client.15174.objecter ms_dispatch 0x7f3084000b90 command_reply(tid 1: 0 ) v1 2024-04-04T21:39:20.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.356+0000 7f30817fa640 10 client.15174.objecter _finish_command 1 = system:0 2024-04-04T21:39:20.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.356+0000 7f30817fa640 20 client.15174.objecter put_session s=0x7f30700744f0 osd=7 4 2024-04-04T21:39:20.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.356+0000 7f30817fa640 15 client.15174.objecter _session_command_op_remove 7 1 2024-04-04T21:39:20.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.369+0000 7f308ac14640 10 client.15174.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-04T21:39:20.371 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.369+0000 7f308ac14640 15 client.15174.objecter _session_command_op_assign -1 2 2024-04-04T21:39:20.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.369+0000 7f308ac14640 20 client.15174.objecter _get_session s=0x7f30700744f0 osd=7 3 2024-04-04T21:39:20.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.369+0000 7f308ac14640 20 client.15174.objecter put_session s=0x7f30700744f0 osd=7 3 2024-04-04T21:39:20.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.369+0000 7f308ac14640 20 client.15174.objecter _get_session s=0x7f30700744f0 osd=7 3 2024-04-04T21:39:20.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.369+0000 7f308ac14640 15 client.15174.objecter _session_command_op_remove -1 2 2024-04-04T21:39:20.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.369+0000 7f308ac14640 20 client.15174.objecter get_session s=0x7f30700744f0 osd=7 3 2024-04-04T21:39:20.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.369+0000 7f308ac14640 15 client.15174.objecter _session_command_op_assign 7 2 2024-04-04T21:39:20.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.369+0000 7f308ac14640 20 client.15174.objecter put_session s=0x7f30700744f0 osd=7 4 2024-04-04T21:39:20.372 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.369+0000 7f308ac14640 10 client.15174.objecter _send_command 2 2024-04-04T21:39:20.373 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.369+0000 7f308ac14640 1 -- 172.21.15.106:0/2399518757 --> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f30841cdb70 con 0x7f3070074bc0 2024-04-04T21:39:20.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.383+0000 7f30817fa640 1 -- 172.21.15.106:0/2399518757 <== osd.7 v2:172.21.15.203:6808/4104640840 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f30841cdb70 con 0x7f3070074bc0 2024-04-04T21:39:20.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.383+0000 7f30817fa640 10 client.15174.objecter ms_dispatch 0x7f3084000b90 command_reply(tid 2: 0 ) v1 2024-04-04T21:39:20.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.383+0000 7f30817fa640 10 client.15174.objecter _finish_command 2 = system:0 2024-04-04T21:39:20.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.383+0000 7f30817fa640 20 client.15174.objecter put_session s=0x7f30700744f0 osd=7 4 2024-04-04T21:39:20.384 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.383+0000 7f30817fa640 15 client.15174.objecter _session_command_op_remove 7 2 2024-04-04T21:39:20.384 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-04T21:39:20.385 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-04T21:39:20.385 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-04T21:39:20.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.383+0000 7f308ac14640 10 librados: watch_flush enter 2024-04-04T21:39:20.386 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 10 librados: watch_flush exit 2024-04-04T21:39:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 10 client.15174.objecter close_session for osd.7 2024-04-04T21:39:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 1 -- 172.21.15.106:0/2399518757 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x7f3070074bc0 msgr2=0x7f3070077060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 1 --2- 172.21.15.106:0/2399518757 >> [v2:172.21.15.203:6808/4104640840,v1:172.21.15.203:6809/4104640840] conn(0x7f3070074bc0 0x7f3070077060 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 20 client.15174.objecter put_session s=0x7f30700744f0 osd=7 1 2024-04-04T21:39:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 20 client.15174.objecter shutdown clearing up homeless session... 2024-04-04T21:39:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 10 client.15174.objecter successfully canceled tick 2024-04-04T21:39:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 1 -- 172.21.15.106:0/2399518757 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f307006ceb0 msgr2=0x7f307006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 1 --2- 172.21.15.106:0/2399518757 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f307006ceb0 0x7f307006f370 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3078009560 tx=0x7f3078009190 comp rx=0 tx=0).stop 2024-04-04T21:39:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 10 monclient: shutdown 2024-04-04T21:39:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 1 -- 172.21.15.106:0/2399518757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30840b5360 msgr2=0x7f30841cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:20.387 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 1 --2- 172.21.15.106:0/2399518757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30840b5360 0x7f30841cf440 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 crypto rx=0x7f307400c440 tx=0x7f307400c900 comp rx=0 tx=0).stop 2024-04-04T21:39:20.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f30891b1640 1 -- 172.21.15.106:0/2399518757 reap_dead start 2024-04-04T21:39:20.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 1 -- 172.21.15.106:0/2399518757 shutdown_connections 2024-04-04T21:39:20.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 1 -- 172.21.15.106:0/2399518757 >> 172.21.15.106:0/2399518757 conn(0x7f3084066e70 msgr2=0x7f3084067250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:20.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 1 -- 172.21.15.106:0/2399518757 shutdown_connections 2024-04-04T21:39:20.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 1 -- 172.21.15.106:0/2399518757 wait complete. 2024-04-04T21:39:20.388 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.384+0000 7f308ac14640 1 librados: shutdown 2024-04-04T21:39:20.396 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-04T21:39:20.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.511+0000 7f8f44d47640 10 monclient: get_monmap_and_config 2024-04-04T21:39:20.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.511+0000 7f8f44d47640 10 monclient: build_initial_monmap 2024-04-04T21:39:20.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.511+0000 7f8f44d47640 10 monclient: monmap: 2024-04-04T21:39:20.512 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:20.512 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:20.512 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:20.512347+0000 2024-04-04T21:39:20.512 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:20.512347+0000 2024-04-04T21:39:20.512 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:20.512 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:20.512 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:20.512 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:20.512 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:20.513 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:20.513 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:20.513 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:20.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 1 Processor -- start 2024-04-04T21:39:20.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 1 -- start start 2024-04-04T21:39:20.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 10 monclient: init 2024-04-04T21:39:20.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:20.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-04-04T21:39:20.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8f400a9a50 0x7f8f400a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:20.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 10 monclient(hunting): picked mon.noname-a con 0x7f8f400a9a50 addr v2:172.21.15.106:3300/0 2024-04-04T21:39:20.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f4013f6a0 0x7f8f4013fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:20.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 10 monclient(hunting): picked mon.noname-b con 0x7f8f4013f6a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:20.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f40142880 addr v1:172.21.15.203:6789/0 2024-04-04T21:39:20.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f40143420 con 0x7f8f40142880 2024-04-04T21:39:20.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:20.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f400aa370 con 0x7f8f400a9a50 2024-04-04T21:39:20.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:20.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400ab750 con 0x7f8f4013f6a0 2024-04-04T21:39:20.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:20.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.512+0000 7f8f44d47640 10 monclient(hunting): authenticate will time out at 3609.779821s 2024-04-04T21:39:20.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.513+0000 7f8f37fff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f4013f6a0 0x7f8f4013fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:20.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.513+0000 7f8f3ed9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f8f40142880 0x7f8f40142d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.106:55746/0 (socket says 172.21.15.106:55746) 2024-04-04T21:39:20.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.513+0000 7f8f3e59c640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8f400a9a50 0x7f8f400a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:20.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.513+0000 7f8f3ed9d640 1 -- 172.21.15.106:0/3098067918 learned_addr learned my addr 172.21.15.106:0/3098067918 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:39:20.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.513+0000 7f8f3e59c640 10 monclient(hunting): get_auth_request con 0x7f8f400a9a50 auth_method 0 2024-04-04T21:39:20.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.513+0000 7f8f3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:20.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.513+0000 7f8f3e59c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:20.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.513+0000 7f8f3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:20.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.513+0000 7f8f37fff640 10 monclient(hunting): get_auth_request con 0x7f8f4013f6a0 auth_method 0 2024-04-04T21:39:20.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.513+0000 7f8f37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:20.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.513+0000 7f8f37fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:20.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.513+0000 7f8f37fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:20.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.514+0000 7f8f3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:20.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.514+0000 7f8f3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:20.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.514+0000 7f8f3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:20.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.514+0000 7f8f37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:20.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.514+0000 7f8f37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:20.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.514+0000 7f8f37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:20.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3dd9b640 1 -- 172.21.15.106:0/3098067918 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277905733 0 0) 0x7f8f40143420 con 0x7f8f40142880 2024-04-04T21:39:20.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3dd9b640 10 monclient(hunting): _negotiate 2024-04-04T21:39:20.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:20.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:20.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3dd9b640 10 monclient(hunting): my global_id is 22132 2024-04-04T21:39:20.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3dd9b640 1 -- 172.21.15.106:0/3098067918 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f2c003520 con 0x7f8f40142880 2024-04-04T21:39:20.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3e59c640 10 monclient(hunting): handle_auth_done global_id 15177 payload 274 2024-04-04T21:39:20.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3e59c640 1 -- 172.21.15.106:0/3098067918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f4013f6a0 msgr2=0x7f8f4013fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:20.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3e59c640 1 --2- 172.21.15.106:0/3098067918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f4013f6a0 0x7f8f4013fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3e59c640 1 -- 172.21.15.106:0/3098067918 >> v1:172.21.15.203:6789/0 conn(0x7f8f40142880 legacy=0x7f8f40142d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:20.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3e59c640 10 monclient: _finish_hunting 0 2024-04-04T21:39:20.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3e59c640 1 monclient: found mon.noname-a 2024-04-04T21:39:20.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:20.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:39:20.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f3e59c640 1 -- 172.21.15.106:0/3098067918 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f40140120 con 0x7f8f400a9a50 2024-04-04T21:39:20.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.515+0000 7f8f37fff640 1 --2- 172.21.15.106:0/3098067918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f4013f6a0 0x7f8f4013fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-04T21:39:20.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.517+0000 7f8f3e59c640 1 --2- 172.21.15.106:0/3098067918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f400a9a50 0x7f8f400a9e30 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 crypto rx=0x7f8f300097a0 tx=0x7f8f3002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b51dbeee9358fd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:20.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f3dd9b640 1 -- 172.21.15.106:0/3098067918 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f300315a0 con 0x7f8f400a9a50 2024-04-04T21:39:20.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f3dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:20.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f3dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:39:20.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f3dd9b640 10 monclient: dump: 2024-04-04T21:39:20.522 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:20.522 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:20.522 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.522 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.522 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:20.523 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f3dd9b640 10 monclient: _finish_auth 0 2024-04-04T21:39:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f3dd9b640 10 monclient: _check_auth_tickets 2024-04-04T21:39:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f3dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f3dd9b640 1 -- 172.21.15.106:0/3098067918 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f30037070 con 0x7f8f400a9a50 2024-04-04T21:39:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f44d47640 5 monclient: authenticate success, global_id 15177 2024-04-04T21:39:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f44d47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:39:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f3dd9b640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:20.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f44d47640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f3dd9b640 1 -- 172.21.15.106:0/3098067918 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f30031960 con 0x7f8f400a9a50 2024-04-04T21:39:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f44d47640 4 set_mon_vals no callback set 2024-04-04T21:39:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f3dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f3dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:39:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.521+0000 7f8f3dd9b640 10 monclient: dump: 2024-04-04T21:39:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:20.524 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:20.525 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:20.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.522+0000 7f8f44d47640 10 monclient: shutdown 2024-04-04T21:39:20.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.522+0000 7f8f44d47640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:20.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.522+0000 7f8f44d47640 1 -- 172.21.15.106:0/3098067918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f400a9a50 msgr2=0x7f8f400a9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:20.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.522+0000 7f8f44d47640 1 --2- 172.21.15.106:0/3098067918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f400a9a50 0x7f8f400a9e30 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 crypto rx=0x7f8f300097a0 tx=0x7f8f3002b840 comp rx=0 tx=0).stop 2024-04-04T21:39:20.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.522+0000 7f8f44d47640 1 -- 172.21.15.106:0/3098067918 shutdown_connections 2024-04-04T21:39:20.528 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.522+0000 7f8f44d47640 1 --2- 172.21.15.106:0/3098067918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f4013f6a0 0x7f8f4013fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.522+0000 7f8f44d47640 1 --2- 172.21.15.106:0/3098067918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f400a9a50 0x7f8f400a9e30 unknown :-1 s=CLOSED pgs=3171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.522+0000 7f8f44d47640 1 -- 172.21.15.106:0/3098067918 >> 172.21.15.106:0/3098067918 conn(0x7f8f400b8030 msgr2=0x7f8f400b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:20.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.522+0000 7f8f44d47640 1 -- 172.21.15.106:0/3098067918 shutdown_connections 2024-04-04T21:39:20.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.522+0000 7f8f44d47640 1 -- 172.21.15.106:0/3098067918 wait complete. 2024-04-04T21:39:20.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.523+0000 7f8f44d47640 10 monclient: build_initial_monmap 2024-04-04T21:39:20.529 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.523+0000 7f8f44d47640 10 monclient: monmap: 2024-04-04T21:39:20.529 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:20.529 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:20.529 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:20.529 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:20.530 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:20.530 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:20.530 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:20.530 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:20.530 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:20.530 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:20.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.523+0000 7f8f44d47640 1 librados: starting msgr at 2024-04-04T21:39:20.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.523+0000 7f8f44d47640 1 librados: starting objecter 2024-04-04T21:39:20.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.523+0000 7f8f44d47640 1 Processor -- start 2024-04-04T21:39:20.530 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.523+0000 7f8f44d47640 1 -- start start 2024-04-04T21:39:20.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.523+0000 7f8f44d47640 1 librados: setting wanted keys 2024-04-04T21:39:20.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.523+0000 7f8f44d47640 1 librados: calling monclient init 2024-04-04T21:39:20.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.523+0000 7f8f44d47640 10 monclient: init 2024-04-04T21:39:20.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:20.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-04T21:39:20.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f400a9a50 0x7f8f40145f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:20.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 10 monclient(hunting): picked mon.noname-b con 0x7f8f400a9a50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:20.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f4013f6a0 0x7f8f40138990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:20.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 10 monclient(hunting): picked mon.noname-a con 0x7f8f4013f6a0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:20.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f40142880 0x7f8f40143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:20.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 10 monclient(hunting): picked mon.noname-c con 0x7f8f40142880 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:20.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:20.531 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4014d070 con 0x7f8f4013f6a0 2024-04-04T21:39:20.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:20.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4014cef0 con 0x7f8f400a9a50 2024-04-04T21:39:20.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:20.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4014d1f0 con 0x7f8f40142880 2024-04-04T21:39:20.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:20.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f44d47640 10 monclient(hunting): authenticate will time out at 3609.791296s 2024-04-04T21:39:20.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f37fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f4013f6a0 0x7f8f40138990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:20.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f3e59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f400a9a50 0x7f8f40145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:20.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f3ed9d640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f40142880 0x7f8f40143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:20.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f37fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f4013f6a0 0x7f8f40138990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42592/0 (socket says 172.21.15.106:42592) 2024-04-04T21:39:20.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f37fff640 1 -- 172.21.15.106:0/2733318531 learned_addr learned my addr 172.21.15.106:0/2733318531 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:20.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f37fff640 10 monclient(hunting): get_auth_request con 0x7f8f4013f6a0 auth_method 0 2024-04-04T21:39:20.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:20.532 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f37fff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:20.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.524+0000 7f8f37fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:20.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f3ed9d640 10 monclient(hunting): get_auth_request con 0x7f8f40142880 auth_method 0 2024-04-04T21:39:20.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f3ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:20.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f3ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:20.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f3ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:20.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f3e59c640 10 monclient(hunting): get_auth_request con 0x7f8f400a9a50 auth_method 0 2024-04-04T21:39:20.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:20.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f3e59c640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:20.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:20.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:20.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:20.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:20.533 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:20.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:20.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:20.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f3ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:20.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f3ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:20.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.525+0000 7f8f3ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:20.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.528+0000 7f8f3e59c640 10 monclient(hunting): handle_auth_done global_id 22135 payload 824 2024-04-04T21:39:20.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.528+0000 7f8f3e59c640 1 -- 172.21.15.106:0/2733318531 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f40142880 msgr2=0x7f8f40143d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:20.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.528+0000 7f8f3e59c640 1 --2- 172.21.15.106:0/2733318531 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f40142880 0x7f8f40143d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.528+0000 7f8f3e59c640 1 -- 172.21.15.106:0/2733318531 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f4013f6a0 msgr2=0x7f8f40138990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:20.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.528+0000 7f8f3e59c640 1 --2- 172.21.15.106:0/2733318531 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f4013f6a0 0x7f8f40138990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.528+0000 7f8f3e59c640 10 monclient: _finish_hunting 0 2024-04-04T21:39:20.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.528+0000 7f8f3e59c640 1 monclient: found mon.noname-b 2024-04-04T21:39:20.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.528+0000 7f8f3e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:20.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.528+0000 7f8f3e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:20.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.528+0000 7f8f3e59c640 1 -- 172.21.15.106:0/2733318531 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f40144f80 con 0x7f8f400a9a50 2024-04-04T21:39:20.536 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.530+0000 7f8f3e59c640 1 --2- 172.21.15.106:0/2733318531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f400a9a50 0x7f8f40145f40 secure :-1 s=READY pgs=5555 cs=0 l=1 rev1=1 crypto rx=0x7f8f400b8fb0 tx=0x7f8f300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:20.536 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.530+0000 7f8f36ffd640 10 client.?.objecter ms_handle_connect 0x7f8f400a9a50 2024-04-04T21:39:20.536 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.530+0000 7f8f36ffd640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:20.536 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f36ffd640 1 -- 172.21.15.106:0/2733318531 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f30045640 con 0x7f8f400a9a50 2024-04-04T21:39:20.536 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:20.536 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f36ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:20.536 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f36ffd640 10 monclient: dump: 2024-04-04T21:39:20.537 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:20.537 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:20.537 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.537 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.537 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:20.537 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:20.537 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:20.537 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:20.537 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:20.537 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:20.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f36ffd640 10 monclient: _finish_auth 0 2024-04-04T21:39:20.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f36ffd640 10 monclient: _check_auth_tickets 2024-04-04T21:39:20.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:20.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f36ffd640 1 -- 172.21.15.106:0/2733318531 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f30045800 con 0x7f8f400a9a50 2024-04-04T21:39:20.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f44d47640 5 monclient: authenticate success, global_id 22135 2024-04-04T21:39:20.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f44d47640 10 monclient: _renew_subs 2024-04-04T21:39:20.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f44d47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:20.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f44d47640 1 -- 172.21.15.106:0/2733318531 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f401451f0 con 0x7f8f400a9a50 2024-04-04T21:39:20.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f36ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:20.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f36ffd640 1 -- 172.21.15.106:0/2733318531 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f30045bf0 con 0x7f8f400a9a50 2024-04-04T21:39:20.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:20.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f36ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:20.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f36ffd640 10 monclient: dump: 2024-04-04T21:39:20.538 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:20.539 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:20.539 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.539 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:20.539 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:20.539 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:20.539 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:20.539 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:20.539 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:20.539 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:20.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f44d47640 10 client.22135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:20.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f44d47640 10 monclient: _renew_subs 2024-04-04T21:39:20.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f44d47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:20.539 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f3d59a640 4 set_mon_vals no callback set 2024-04-04T21:39:20.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f44d47640 1 -- 172.21.15.106:0/2733318531 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f40135f80 con 0x7f8f400a9a50 2024-04-04T21:39:20.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.534+0000 7f8f44d47640 1 librados: init done 2024-04-04T21:39:20.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.535+0000 7f8f44d47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:20.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.535+0000 7f8f44d47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:20.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.535+0000 7f8f44d47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:20.540 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.535+0000 7f8f44d47640 1 -- 172.21.15.106:0/2733318531 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f40138ed0 con 0x7f8f400a9a50 2024-04-04T21:39:20.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.781+0000 7f8f36ffd640 1 -- 172.21.15.106:0/2733318531 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8f3004c370 con 0x7f8f400a9a50 2024-04-04T21:39:20.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.781+0000 7f8f36ffd640 1 --2- 172.21.15.106:0/2733318531 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f8f2006ceb0 0x7f8f2006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:20.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.781+0000 7f8f37fff640 1 --2- 172.21.15.106:0/2733318531 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f8f2006ceb0 0x7f8f2006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:20.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.781+0000 7f8f37fff640 10 monclient: get_auth_request con 0x7f8f2006ceb0 auth_method 0 2024-04-04T21:39:20.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.782+0000 7f8f37fff640 1 --2- 172.21.15.106:0/2733318531 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f8f2006ceb0 0x7f8f2006f370 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009560 tx=0x7f8f28009190 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:20.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.782+0000 7f8f36ffd640 10 client.22135.objecter ms_handle_connect 0x7f8f2006ceb0 2024-04-04T21:39:20.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.783+0000 7f8f36ffd640 1 -- 172.21.15.106:0/2733318531 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f8f3002f020 con 0x7f8f400a9a50 2024-04-04T21:39:20.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.783+0000 7f8f36ffd640 10 client.22135.objecter ms_dispatch 0x7f8f40000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:20.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.783+0000 7f8f36ffd640 3 client.22135.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:20.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.783+0000 7f8f36ffd640 3 client.22135.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:20.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.783+0000 7f8f36ffd640 20 client.22135.objecter dump_active .. 0 homeless 2024-04-04T21:39:20.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.793+0000 7f8f36ffd640 1 -- 172.21.15.106:0/2733318531 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8f40138ed0 con 0x7f8f400a9a50 2024-04-04T21:39:20.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.793+0000 7f8f36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:20.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.793+0000 7f8f36ffd640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:20.919 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:12.433 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:20.968 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:12.485 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:20.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.974+0000 7f8f44d47640 1 -- 172.21.15.106:0/2733318531 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f4004f050 con 0x7f8f2006ceb0 2024-04-04T21:39:20.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.974+0000 7f8f36ffd640 1 -- 172.21.15.106:0/2733318531 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8f4004f050 con 0x7f8f2006ceb0 2024-04-04T21:39:20.975 INFO:teuthology.orchestra.run.smithi106.stderr:instructed osd(s) 7 to deep-scrub 2024-04-04T21:39:20.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.976+0000 7f8f44d47640 10 librados: watch_flush enter 2024-04-04T21:39:20.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.976+0000 7f8f44d47640 10 librados: watch_flush exit 2024-04-04T21:39:20.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.976+0000 7f8f44d47640 20 client.22135.objecter shutdown clearing up homeless session... 2024-04-04T21:39:20.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.976+0000 7f8f44d47640 10 client.22135.objecter successfully canceled tick 2024-04-04T21:39:20.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.976+0000 7f8f44d47640 1 -- 172.21.15.106:0/2733318531 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f8f2006ceb0 msgr2=0x7f8f2006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:20.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.976+0000 7f8f44d47640 1 --2- 172.21.15.106:0/2733318531 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f8f2006ceb0 0x7f8f2006f370 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009560 tx=0x7f8f28009190 comp rx=0 tx=0).stop 2024-04-04T21:39:20.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.976+0000 7f8f44d47640 10 monclient: shutdown 2024-04-04T21:39:20.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.976+0000 7f8f44d47640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:20.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.976+0000 7f8f44d47640 1 -- 172.21.15.106:0/2733318531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f400a9a50 msgr2=0x7f8f40145f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:20.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.976+0000 7f8f44d47640 1 --2- 172.21.15.106:0/2733318531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f400a9a50 0x7f8f40145f40 secure :-1 s=READY pgs=5555 cs=0 l=1 rev1=1 crypto rx=0x7f8f400b8fb0 tx=0x7f8f300046c0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.977+0000 7f8f44d47640 1 -- 172.21.15.106:0/2733318531 shutdown_connections 2024-04-04T21:39:20.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.977+0000 7f8f44d47640 1 --2- 172.21.15.106:0/2733318531 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f8f2006ceb0 0x7f8f2006f370 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.977+0000 7f8f44d47640 1 --2- 172.21.15.106:0/2733318531 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f40142880 0x7f8f40143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.978 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.977+0000 7f8f44d47640 1 --2- 172.21.15.106:0/2733318531 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f4013f6a0 0x7f8f40138990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.977+0000 7f8f44d47640 1 --2- 172.21.15.106:0/2733318531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f400a9a50 0x7f8f40145f40 unknown :-1 s=CLOSED pgs=5555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:20.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.977+0000 7f8f44d47640 1 -- 172.21.15.106:0/2733318531 >> 172.21.15.106:0/2733318531 conn(0x7f8f400b8030 msgr2=0x7f8f400b8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:20.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.977+0000 7f8f44d47640 1 -- 172.21.15.106:0/2733318531 shutdown_connections 2024-04-04T21:39:20.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.977+0000 7f8f44d47640 1 -- 172.21.15.106:0/2733318531 wait complete. 2024-04-04T21:39:20.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:20.977+0000 7f8f44d47640 1 librados: shutdown 2024-04-04T21:39:20.985 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:12.502 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:20.991 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-04T21:39:21.023 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:12.505 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:21.026 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:12.508 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:21.029 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:12.546 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:21.032 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:12.549 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:21.040 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:12.523 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:21.048 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:12.564 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:21.111 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.111+0000 7f42d740a640 10 monclient: get_monmap_and_config 2024-04-04T21:39:21.111 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.111+0000 7f42d740a640 10 monclient: build_initial_monmap 2024-04-04T21:39:21.111 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.111+0000 7f42d740a640 10 monclient: monmap: 2024-04-04T21:39:21.111 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:21.111 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:21.111 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:21.112078+0000 2024-04-04T21:39:21.111 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:21.112078+0000 2024-04-04T21:39:21.112 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:21.112 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:21.112 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:21.112 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:21.112 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:21.112 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:21.112 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:21.112 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:21.112 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.111+0000 7f42d740a640 1 Processor -- start 2024-04-04T21:39:21.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.111+0000 7f42d740a640 1 -- start start 2024-04-04T21:39:21.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.111+0000 7f42d740a640 10 monclient: init 2024-04-04T21:39:21.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.112+0000 7f42d740a640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:21.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.112+0000 7f42d740a640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-04T21:39:21.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.112+0000 7f42d740a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42d00a9b60 addr v1:172.21.15.203:6789/0 2024-04-04T21:39:21.113 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.112+0000 7f42d740a640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f42d01426f0 0x7f42d00b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:21.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.112+0000 7f42d740a640 10 monclient(hunting): picked mon.noname-a con 0x7f42d01426f0 addr v2:172.21.15.106:3300/0 2024-04-04T21:39:21.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.112+0000 7f42d740a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42d013f810 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:21.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.112+0000 7f42d740a640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d00b2e00 con 0x7f42d013f810 2024-04-04T21:39:21.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.112+0000 7f42d740a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d00b8c90 con 0x7f42d00a9b60 2024-04-04T21:39:21.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.112+0000 7f42d740a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:21.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.112+0000 7f42d740a640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42d00a9f40 con 0x7f42d01426f0 2024-04-04T21:39:21.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.112+0000 7f42d740a640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:21.114 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.112+0000 7f42d740a640 10 monclient(hunting): authenticate will time out at 3610.379543s 2024-04-04T21:39:21.115 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.115+0000 7f42d49a5640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f42d01426f0 0x7f42d00b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:21.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.129+0000 7f42d49a5640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f42d01426f0 0x7f42d00b28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42596/0 (socket says 172.21.15.106:42596) 2024-04-04T21:39:21.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.129+0000 7f42d49a5640 1 -- 172.21.15.106:0/2460541523 learned_addr learned my addr 172.21.15.106:0/2460541523 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:21.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.129+0000 7f42d49a5640 10 monclient(hunting): get_auth_request con 0x7f42d01426f0 auth_method 0 2024-04-04T21:39:21.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.129+0000 7f42d49a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:21.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.129+0000 7f42d49a5640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:21.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.129+0000 7f42d49a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:21.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.131+0000 7f42d49a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:21.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.131+0000 7f42d49a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:21.131 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.131+0000 7f42d49a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:21.139 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.139+0000 7f42d49a5640 10 monclient(hunting): handle_auth_done global_id 15183 payload 274 2024-04-04T21:39:21.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.139+0000 7f42d49a5640 1 -- 172.21.15.106:0/2460541523 >> v1:172.21.15.203:6789/0 conn(0x7f42d00a9b60 legacy=0x7f42d01430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:21.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.139+0000 7f42d49a5640 1 -- 172.21.15.106:0/2460541523 >> v1:172.21.15.106:6789/0 conn(0x7f42d013f810 legacy=0x7f42d013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:21.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.139+0000 7f42d49a5640 10 monclient: _finish_hunting 0 2024-04-04T21:39:21.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.139+0000 7f42d49a5640 1 monclient: found mon.noname-a 2024-04-04T21:39:21.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.139+0000 7f42d49a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:21.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.139+0000 7f42d49a5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.106:3300/0 2024-04-04T21:39:21.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.139+0000 7f42d49a5640 1 -- 172.21.15.106:0/2460541523 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d00b23b0 con 0x7f42d01426f0 2024-04-04T21:39:21.166 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.166+0000 7f42d49a5640 1 --2- 172.21.15.106:0/2460541523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42d01426f0 0x7f42d00b28c0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009870 tx=0x7f42c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c10be01fe1223e1c server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:21.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42c7fff640 1 -- 172.21.15.106:0/2460541523 <== mon.0 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c0035020 con 0x7f42d01426f0 2024-04-04T21:39:21.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42c7fff640 10 monclient: dump: 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42c7fff640 10 monclient: _finish_auth 0 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42c7fff640 10 monclient: _check_auth_tickets 2024-04-04T21:39:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:21.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42c7fff640 1 -- 172.21.15.106:0/2460541523 <== mon.0 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c0031870 con 0x7f42d01426f0 2024-04-04T21:39:21.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42d740a640 5 monclient: authenticate success, global_id 15183 2024-04-04T21:39:21.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42d740a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-04T21:39:21.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42c7fff640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:21.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42d740a640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:21.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42c7fff640 1 -- 172.21.15.106:0/2460541523 <== mon.0 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c0031c40 con 0x7f42d01426f0 2024-04-04T21:39:21.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42d740a640 4 set_mon_vals no callback set 2024-04-04T21:39:21.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:21.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-04T21:39:21.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.180+0000 7f42c7fff640 10 monclient: dump: 2024-04-04T21:39:21.182 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:21.182 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:21.182 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:21.183 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:21.183 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:21.183 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:21.183 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:21.183 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:21.183 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:21.183 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:21.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.181+0000 7f42d740a640 10 monclient: shutdown 2024-04-04T21:39:21.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.181+0000 7f42d740a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:21.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.181+0000 7f42d740a640 1 -- 172.21.15.106:0/2460541523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42d01426f0 msgr2=0x7f42d00b28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:21.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.181+0000 7f42d740a640 1 --2- 172.21.15.106:0/2460541523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42d01426f0 0x7f42d00b28c0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009870 tx=0x7f42c002b910 comp rx=0 tx=0).stop 2024-04-04T21:39:21.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.181+0000 7f42d740a640 1 -- 172.21.15.106:0/2460541523 shutdown_connections 2024-04-04T21:39:21.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.181+0000 7f42d740a640 1 --2- 172.21.15.106:0/2460541523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42d01426f0 0x7f42d00b28c0 unknown :-1 s=CLOSED pgs=3172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:21.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.181+0000 7f42d740a640 1 -- 172.21.15.106:0/2460541523 >> 172.21.15.106:0/2460541523 conn(0x7f42d00b7ec0 msgr2=0x7f42d00b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:21.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.181+0000 7f42d740a640 1 -- 172.21.15.106:0/2460541523 shutdown_connections 2024-04-04T21:39:21.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.181+0000 7f42d740a640 1 -- 172.21.15.106:0/2460541523 wait complete. 2024-04-04T21:39:21.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 10 monclient: build_initial_monmap 2024-04-04T21:39:21.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 10 monclient: monmap: 2024-04-04T21:39:21.186 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:21.186 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:21.186 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:21.186 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:21.186 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:21.186 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:21.186 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:21.186 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:21.186 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:21.186 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:21.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 1 librados: starting msgr at 2024-04-04T21:39:21.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 1 librados: starting objecter 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 1 Processor -- start 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 1 -- start start 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 1 librados: setting wanted keys 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 1 librados: calling monclient init 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 10 monclient: init 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42d00a9b60 0x7f42d013ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 10 monclient(hunting): picked mon.noname-c con 0x7f42d00a9b60 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42d013f810 0x7f42d0139e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 10 monclient(hunting): picked mon.noname-b con 0x7f42d013f810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42d00abf60 0x7f42d0137300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 10 monclient(hunting): picked mon.noname-a con 0x7f42d00abf60 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d014e220 con 0x7f42d00abf60 2024-04-04T21:39:21.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:21.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d0142c10 con 0x7f42d013f810 2024-04-04T21:39:21.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:21.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d014e3a0 con 0x7f42d00a9b60 2024-04-04T21:39:21.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:21.188 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.182+0000 7f42d740a640 10 monclient(hunting): authenticate will time out at 3610.450116s 2024-04-04T21:39:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.185+0000 7f42d59a7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42d00abf60 0x7f42d0137300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.186+0000 7f42d51a6640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42d00a9b60 0x7f42d013ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.186+0000 7f42d59a7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42d00abf60 0x7f42d0137300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:42610/0 (socket says 172.21.15.106:42610) 2024-04-04T21:39:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.186+0000 7f42d59a7640 1 -- 172.21.15.106:0/618260789 learned_addr learned my addr 172.21.15.106:0/618260789 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.186+0000 7f42d59a7640 10 monclient(hunting): get_auth_request con 0x7f42d00abf60 auth_method 0 2024-04-04T21:39:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.186+0000 7f42d59a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.186+0000 7f42d59a7640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.186+0000 7f42d59a7640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:21.191 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.191+0000 7f42d49a5640 1 --2- 172.21.15.106:0/618260789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42d013f810 0x7f42d0139e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:21.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.196+0000 7f42d49a5640 10 monclient(hunting): get_auth_request con 0x7f42d013f810 auth_method 0 2024-04-04T21:39:21.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.196+0000 7f42d49a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:21.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.196+0000 7f42d49a5640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:21.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.196+0000 7f42d49a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:21.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.199+0000 7f42d49a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:21.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.199+0000 7f42d49a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:21.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.199+0000 7f42d49a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:21.202 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.203+0000 7f42d51a6640 10 monclient(hunting): get_auth_request con 0x7f42d00a9b60 auth_method 0 2024-04-04T21:39:21.203 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.203+0000 7f42d51a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:21.203 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.203+0000 7f42d51a6640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:21.203 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.203+0000 7f42d51a6640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:21.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.215+0000 7f42d49a5640 10 monclient(hunting): handle_auth_done global_id 22141 payload 824 2024-04-04T21:39:21.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.215+0000 7f42d49a5640 1 -- 172.21.15.106:0/618260789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42d00a9b60 msgr2=0x7f42d013ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:21.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.215+0000 7f42d49a5640 1 --2- 172.21.15.106:0/618260789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42d00a9b60 0x7f42d013ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:21.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.215+0000 7f42d49a5640 1 -- 172.21.15.106:0/618260789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42d00abf60 msgr2=0x7f42d0137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:21.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.215+0000 7f42d49a5640 1 --2- 172.21.15.106:0/618260789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42d00abf60 0x7f42d0137300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:21.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.215+0000 7f42d49a5640 10 monclient: _finish_hunting 0 2024-04-04T21:39:21.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.215+0000 7f42d49a5640 1 monclient: found mon.noname-b 2024-04-04T21:39:21.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.215+0000 7f42d49a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:21.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.215+0000 7f42d49a5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:21.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.215+0000 7f42d49a5640 1 -- 172.21.15.106:0/618260789 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d013bc90 con 0x7f42d013f810 2024-04-04T21:39:21.221 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.221+0000 7f42d49a5640 1 --2- 172.21.15.106:0/618260789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42d013f810 0x7f42d0139e60 secure :-1 s=READY pgs=5557 cs=0 l=1 rev1=1 crypto rx=0x7f42c0033000 tx=0x7f42c00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:21.221 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.221+0000 7f42c5ffb640 10 client.?.objecter ms_handle_connect 0x7f42d013f810 2024-04-04T21:39:21.221 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.221+0000 7f42c5ffb640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:21.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42c5ffb640 1 -- 172.21.15.106:0/618260789 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c00443f0 con 0x7f42d013f810 2024-04-04T21:39:21.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:21.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42c5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:21.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42c5ffb640 10 monclient: dump: 2024-04-04T21:39:21.243 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:21.243 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:21.245 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:21.245 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:21.246 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:21.246 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:21.246 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:21.246 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:21.246 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:21.247 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:21.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42c5ffb640 10 monclient: _finish_auth 0 2024-04-04T21:39:21.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42c5ffb640 10 monclient: _check_auth_tickets 2024-04-04T21:39:21.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:21.247 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42c5ffb640 1 -- 172.21.15.106:0/618260789 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c0044590 con 0x7f42d013f810 2024-04-04T21:39:21.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:21.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42c5ffb640 1 -- 172.21.15.106:0/618260789 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c0044980 con 0x7f42d013f810 2024-04-04T21:39:21.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42d740a640 5 monclient: authenticate success, global_id 22141 2024-04-04T21:39:21.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42d740a640 10 monclient: _renew_subs 2024-04-04T21:39:21.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42d740a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:21.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42d740a640 1 -- 172.21.15.106:0/618260789 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d013bf70 con 0x7f42d013f810 2024-04-04T21:39:21.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42c77fe640 4 set_mon_vals no callback set 2024-04-04T21:39:21.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:21.249 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42c5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:21.249 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42c5ffb640 10 monclient: dump: 2024-04-04T21:39:21.249 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:21.249 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:21.249 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:21.249 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:21.249 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:21.249 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:21.249 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:21.250 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:21.250 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:21.250 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:21.250 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42d740a640 10 client.22141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:21.250 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42d740a640 10 monclient: _renew_subs 2024-04-04T21:39:21.250 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42d740a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:21.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42d740a640 1 -- 172.21.15.106:0/618260789 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d013a3a0 con 0x7f42d013f810 2024-04-04T21:39:21.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.241+0000 7f42d740a640 1 librados: init done 2024-04-04T21:39:21.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.242+0000 7f42d740a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:21.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.242+0000 7f42d740a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:21.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.242+0000 7f42d740a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:21.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.242+0000 7f42d740a640 1 -- 172.21.15.106:0/618260789 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d004f050 con 0x7f42d013f810 2024-04-04T21:39:21.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.579+0000 7f42c5ffb640 1 -- 172.21.15.106:0/618260789 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f42c004b060 con 0x7f42d013f810 2024-04-04T21:39:21.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.579+0000 7f42c5ffb640 1 --2- 172.21.15.106:0/618260789 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f42b406cee0 0x7f42b406f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:21.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.579+0000 7f42d51a6640 1 --2- 172.21.15.106:0/618260789 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f42b406cee0 0x7f42b406f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:21.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.579+0000 7f42d51a6640 10 monclient: get_auth_request con 0x7f42b406cee0 auth_method 0 2024-04-04T21:39:21.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.579+0000 7f42c5ffb640 10 client.22141.objecter ms_handle_connect 0x7f42b406cee0 2024-04-04T21:39:21.580 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.579+0000 7f42d51a6640 1 --2- 172.21.15.106:0/618260789 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f42b406cee0 0x7f42b406f3a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f42cc007fd0 tx=0x7f42cc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:21.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.583+0000 7f42c5ffb640 1 -- 172.21.15.106:0/618260789 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f42c002fe00 con 0x7f42d013f810 2024-04-04T21:39:21.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.583+0000 7f42c5ffb640 10 client.22141.objecter ms_dispatch 0x7f42d0000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:21.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.583+0000 7f42c5ffb640 3 client.22141.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:21.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.583+0000 7f42c5ffb640 3 client.22141.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:21.583 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.583+0000 7f42c5ffb640 20 client.22141.objecter dump_active .. 0 homeless 2024-04-04T21:39:21.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.592+0000 7f42c5ffb640 1 -- 172.21.15.106:0/618260789 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f42c003f070 con 0x7f42d013f810 2024-04-04T21:39:21.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.592+0000 7f42c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:21.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.592+0000 7f42c5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:21.795 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.795+0000 7f42d740a640 1 -- 172.21.15.106:0/618260789 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f42d00b6060 con 0x7f42b406cee0 2024-04-04T21:39:21.798 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.798+0000 7f42c5ffb640 1 -- 172.21.15.106:0/618260789 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+115833 (secure 0 0 0) 0x7f42d00b6060 con 0x7f42b406cee0 2024-04-04T21:39:21.800 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:39:21.800 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-04-04T21:39:21.802 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 10 librados: watch_flush enter 2024-04-04T21:39:21.802 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 10 librados: watch_flush exit 2024-04-04T21:39:21.802 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 20 client.22141.objecter shutdown clearing up homeless session... 2024-04-04T21:39:21.802 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 10 client.22141.objecter successfully canceled tick 2024-04-04T21:39:21.802 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 1 -- 172.21.15.106:0/618260789 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f42b406cee0 msgr2=0x7f42b406f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:21.802 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 1 --2- 172.21.15.106:0/618260789 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f42b406cee0 0x7f42b406f3a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f42cc007fd0 tx=0x7f42cc009040 comp rx=0 tx=0).stop 2024-04-04T21:39:21.802 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 10 monclient: shutdown 2024-04-04T21:39:21.802 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:21.803 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 1 -- 172.21.15.106:0/618260789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42d013f810 msgr2=0x7f42d0139e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:21.803 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 1 --2- 172.21.15.106:0/618260789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42d013f810 0x7f42d0139e60 secure :-1 s=READY pgs=5557 cs=0 l=1 rev1=1 crypto rx=0x7f42c0033000 tx=0x7f42c00046c0 comp rx=0 tx=0).stop 2024-04-04T21:39:21.803 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 1 -- 172.21.15.106:0/618260789 shutdown_connections 2024-04-04T21:39:21.803 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 1 --2- 172.21.15.106:0/618260789 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f42b406cee0 0x7f42b406f3a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:21.803 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 1 --2- 172.21.15.106:0/618260789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42d00abf60 0x7f42d0137300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:21.803 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 1 --2- 172.21.15.106:0/618260789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42d013f810 0x7f42d0139e60 unknown :-1 s=CLOSED pgs=5557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:21.803 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 1 --2- 172.21.15.106:0/618260789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42d00a9b60 0x7f42d013ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:21.803 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 1 -- 172.21.15.106:0/618260789 >> 172.21.15.106:0/618260789 conn(0x7f42d00b7ec0 msgr2=0x7f42d00b7680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:21.803 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 1 -- 172.21.15.106:0/618260789 shutdown_connections 2024-04-04T21:39:21.803 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 1 -- 172.21.15.106:0/618260789 wait complete. 2024-04-04T21:39:21.803 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:21.801+0000 7f42d740a640 1 librados: shutdown 2024-04-04T21:39:21.812 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":2317,"stamp":"2024-04-04T21:39:21.061798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":572584,"kb_used_data":42712,"kb_used_omap":12,"kb_used_meta":529779,"kb_avail":754402136,"statfs":{"total":773094113280,"available":772507787264,"internally_reserved":0,"allocated":43737088,"data_stored":21811050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":542494257},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":92,"apply_latency_ms":92,"commit_latency_ns":92000000,"apply_latency_ns":92000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.108158"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2340,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:10.919685+0000","last_change":"2024-04-04T21:03:54.942652+0000","last_active":"2024-04-04T21:39:10.919685+0000","last_peered":"2024-04-04T21:39:10.919685+0000","last_clean":"2024-04-04T21:39:10.919685+0000","last_became_active":"2024-04-04T21:03:54.933965+0000","last_became_peered":"2024-04-04T21:03:54.933965+0000","last_unstale":"2024-04-04T21:39:10.919685+0000","last_undegraded":"2024-04-04T21:39:10.919685+0000","last_fullsized":"2024-04-04T21:39:10.919685+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:47:55.727002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2311,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.252688+0000","last_change":"2024-04-04T21:03:44.565986+0000","last_active":"2024-04-04T21:39:11.252688+0000","last_peered":"2024-04-04T21:39:11.252688+0000","last_clean":"2024-04-04T21:39:11.252688+0000","last_became_active":"2024-04-04T21:03:44.551107+0000","last_became_peered":"2024-04-04T21:03:44.551107+0000","last_unstale":"2024-04-04T21:39:11.252688+0000","last_undegraded":"2024-04-04T21:39:11.252688+0000","last_fullsized":"2024-04-04T21:39:11.252688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2315,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:20.196589+0000","last_change":"2024-04-04T21:39:20.197387+0000","last_active":"2024-04-04T21:39:20.196589+0000","last_peered":"2024-04-04T21:39:20.196589+0000","last_clean":"2024-04-04T21:39:20.196589+0000","last_became_active":"2024-04-04T21:03:52.772128+0000","last_became_peered":"2024-04-04T21:03:52.772128+0000","last_unstale":"2024-04-04T21:39:20.196589+0000","last_undegraded":"2024-04-04T21:39:20.196589+0000","last_fullsized":"2024-04-04T21:39:20.196589+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:20.195013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:20.195013+0000","last_clean_scrub_stamp":"2024-04-04T21:39:20.195013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:45:43.100947+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2292,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.267093+0000","last_change":"2024-04-04T21:03:41.361551+0000","last_active":"2024-04-04T21:39:11.267093+0000","last_peered":"2024-04-04T21:39:11.267093+0000","last_clean":"2024-04-04T21:39:11.267093+0000","last_became_active":"2024-04-04T21:03:41.346161+0000","last_became_peered":"2024-04-04T21:03:41.346161+0000","last_unstale":"2024-04-04T21:39:11.267093+0000","last_undegraded":"2024-04-04T21:39:11.267093+0000","last_fullsized":"2024-04-04T21:39:11.267093+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:14:45.723197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2295,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.284730+0000","last_change":"2024-04-04T21:03:42.130175+0000","last_active":"2024-04-04T21:39:11.284730+0000","last_peered":"2024-04-04T21:39:11.284730+0000","last_clean":"2024-04-04T21:39:11.284730+0000","last_became_active":"2024-04-04T21:03:42.122180+0000","last_became_peered":"2024-04-04T21:03:42.122180+0000","last_unstale":"2024-04-04T21:39:11.284730+0000","last_undegraded":"2024-04-04T21:39:11.284730+0000","last_fullsized":"2024-04-04T21:39:11.284730+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:01:39.698809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2322,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.235906+0000","last_change":"2024-04-04T21:03:42.949631+0000","last_active":"2024-04-04T21:39:11.235906+0000","last_peered":"2024-04-04T21:39:11.235906+0000","last_clean":"2024-04-04T21:39:11.235906+0000","last_became_active":"2024-04-04T21:03:42.940728+0000","last_became_peered":"2024-04-04T21:03:42.940728+0000","last_unstale":"2024-04-04T21:39:11.235906+0000","last_undegraded":"2024-04-04T21:39:11.235906+0000","last_fullsized":"2024-04-04T21:39:11.235906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:24:58.260325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2311,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.248362+0000","last_change":"2024-04-04T21:03:43.791130+0000","last_active":"2024-04-04T21:39:11.248362+0000","last_peered":"2024-04-04T21:39:11.248362+0000","last_clean":"2024-04-04T21:39:11.248362+0000","last_became_active":"2024-04-04T21:03:43.596620+0000","last_became_peered":"2024-04-04T21:03:43.596620+0000","last_unstale":"2024-04-04T21:39:11.248362+0000","last_undegraded":"2024-04-04T21:39:11.248362+0000","last_fullsized":"2024-04-04T21:39:11.248362+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2313,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.276236+0000","last_change":"2024-04-04T21:03:52.362151+0000","last_active":"2024-04-04T21:39:11.276236+0000","last_peered":"2024-04-04T21:39:11.276236+0000","last_clean":"2024-04-04T21:39:11.276236+0000","last_became_active":"2024-04-04T21:03:52.351392+0000","last_became_peered":"2024-04-04T21:03:52.351392+0000","last_unstale":"2024-04-04T21:39:11.276236+0000","last_undegraded":"2024-04-04T21:39:11.276236+0000","last_fullsized":"2024-04-04T21:39:11.276236+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"39'7","reported_seq":2336,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.213931+0000","last_change":"2024-04-04T21:03:50.565469+0000","last_active":"2024-04-04T21:39:11.213931+0000","last_peered":"2024-04-04T21:39:11.213931+0000","last_clean":"2024-04-04T21:39:11.213931+0000","last_became_active":"2024-04-04T21:03:50.287197+0000","last_became_peered":"2024-04-04T21:03:50.287197+0000","last_unstale":"2024-04-04T21:39:11.213931+0000","last_undegraded":"2024-04-04T21:39:11.213931+0000","last_fullsized":"2024-04-04T21:39:11.213931+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T04:15:05.649671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2295,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.235230+0000","last_change":"2024-04-04T21:03:44.924431+0000","last_active":"2024-04-04T21:39:11.235230+0000","last_peered":"2024-04-04T21:39:11.235230+0000","last_clean":"2024-04-04T21:39:11.235230+0000","last_became_active":"2024-04-04T21:03:44.913048+0000","last_became_peered":"2024-04-04T21:03:44.913048+0000","last_unstale":"2024-04-04T21:39:11.235230+0000","last_undegraded":"2024-04-04T21:39:11.235230+0000","last_fullsized":"2024-04-04T21:39:11.235230+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T22:52:04.454928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2302,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:16.850272+0000","last_change":"2024-04-04T21:39:16.851029+0000","last_active":"2024-04-04T21:39:16.850272+0000","last_peered":"2024-04-04T21:39:16.850272+0000","last_clean":"2024-04-04T21:39:16.850272+0000","last_became_active":"2024-04-04T21:03:44.451112+0000","last_became_peered":"2024-04-04T21:03:44.451112+0000","last_unstale":"2024-04-04T21:39:16.850272+0000","last_undegraded":"2024-04-04T21:39:16.850272+0000","last_fullsized":"2024-04-04T21:39:16.850272+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:16.848683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:16.848683+0000","last_clean_scrub_stamp":"2024-04-04T21:39:16.848683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T09:27:15.312906+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2342,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:10.930152+0000","last_change":"2024-04-04T21:03:44.379999+0000","last_active":"2024-04-04T21:39:10.930152+0000","last_peered":"2024-04-04T21:39:10.930152+0000","last_clean":"2024-04-04T21:39:10.930152+0000","last_became_active":"2024-04-04T21:03:44.366452+0000","last_became_peered":"2024-04-04T21:03:44.366452+0000","last_unstale":"2024-04-04T21:39:10.930152+0000","last_undegraded":"2024-04-04T21:39:10.930152+0000","last_fullsized":"2024-04-04T21:39:10.930152+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:21:39.463272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"1197'78","reported_seq":2333,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:16.155555+0000","last_change":"2024-04-04T21:39:16.157056+0000","last_active":"2024-04-04T21:39:16.155555+0000","last_peered":"2024-04-04T21:39:16.155555+0000","last_clean":"2024-04-04T21:39:16.155555+0000","last_became_active":"2024-04-04T21:02:19.876220+0000","last_became_peered":"2024-04-04T21:02:19.876220+0000","last_unstale":"2024-04-04T21:39:16.155555+0000","last_undegraded":"2024-04-04T21:39:16.155555+0000","last_fullsized":"2024-04-04T21:39:16.155555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"1197'78","last_scrub_stamp":"2024-04-04T21:39:16.150979+0000","last_deep_scrub":"1197'78","last_deep_scrub_stamp":"2024-04-04T21:39:16.150979+0000","last_clean_scrub_stamp":"2024-04-04T21:39:16.150979+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T23:01:55.827963+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2328,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:10.986983+0000","last_change":"2024-04-04T21:02:24.476744+0000","last_active":"2024-04-04T21:39:10.986983+0000","last_peered":"2024-04-04T21:39:10.986983+0000","last_clean":"2024-04-04T21:39:10.986983+0000","last_became_active":"2024-04-04T21:02:24.466203+0000","last_became_peered":"2024-04-04T21:02:24.466203+0000","last_unstale":"2024-04-04T21:39:10.986983+0000","last_undegraded":"2024-04-04T21:39:10.986983+0000","last_fullsized":"2024-04-04T21:39:10.986983+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2294,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.239627+0000","last_change":"2024-04-04T21:03:40.772628+0000","last_active":"2024-04-04T21:39:11.239627+0000","last_peered":"2024-04-04T21:39:11.239627+0000","last_clean":"2024-04-04T21:39:11.239627+0000","last_became_active":"2024-04-04T21:03:40.758544+0000","last_became_peered":"2024-04-04T21:03:40.758544+0000","last_unstale":"2024-04-04T21:39:11.239627+0000","last_undegraded":"2024-04-04T21:39:11.239627+0000","last_fullsized":"2024-04-04T21:39:11.239627+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:22:11.238559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2337,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:18.382155+0000","last_change":"2024-04-04T21:39:18.382966+0000","last_active":"2024-04-04T21:39:18.382155+0000","last_peered":"2024-04-04T21:39:18.382155+0000","last_clean":"2024-04-04T21:39:18.382155+0000","last_became_active":"2024-04-04T21:02:24.146676+0000","last_became_peered":"2024-04-04T21:02:24.146676+0000","last_unstale":"2024-04-04T21:39:18.382155+0000","last_undegraded":"2024-04-04T21:39:18.382155+0000","last_fullsized":"2024-04-04T21:39:18.382155+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:18.379250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:18.379250+0000","last_clean_scrub_stamp":"2024-04-04T21:39:18.379250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:11:16.619775+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2292,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.262624+0000","last_change":"2024-04-04T21:03:39.388042+0000","last_active":"2024-04-04T21:39:11.262624+0000","last_peered":"2024-04-04T21:39:11.262624+0000","last_clean":"2024-04-04T21:39:11.262624+0000","last_became_active":"2024-04-04T21:03:39.382089+0000","last_became_peered":"2024-04-04T21:03:39.382089+0000","last_unstale":"2024-04-04T21:39:11.262624+0000","last_undegraded":"2024-04-04T21:39:11.262624+0000","last_fullsized":"2024-04-04T21:39:11.262624+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T01:47:50.490851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2334,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.217618+0000","last_change":"2024-04-04T21:02:24.189161+0000","last_active":"2024-04-04T21:39:11.217618+0000","last_peered":"2024-04-04T21:39:11.217618+0000","last_clean":"2024-04-04T21:39:11.217618+0000","last_became_active":"2024-04-04T21:02:24.159102+0000","last_became_peered":"2024-04-04T21:02:24.159102+0000","last_unstale":"2024-04-04T21:39:11.217618+0000","last_undegraded":"2024-04-04T21:39:11.217618+0000","last_fullsized":"2024-04-04T21:39:11.217618+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:02:24.748876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2301,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.018185+0000","last_change":"2024-04-04T21:03:45.245420+0000","last_active":"2024-04-04T21:39:11.018185+0000","last_peered":"2024-04-04T21:39:11.018185+0000","last_clean":"2024-04-04T21:39:11.018185+0000","last_became_active":"2024-04-04T21:03:45.238093+0000","last_became_peered":"2024-04-04T21:03:45.238093+0000","last_unstale":"2024-04-04T21:39:11.018185+0000","last_undegraded":"2024-04-04T21:39:11.018185+0000","last_fullsized":"2024-04-04T21:39:11.018185+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2328,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:10.982667+0000","last_change":"2024-04-04T21:02:24.359272+0000","last_active":"2024-04-04T21:39:10.982667+0000","last_peered":"2024-04-04T21:39:10.982667+0000","last_clean":"2024-04-04T21:39:10.982667+0000","last_became_active":"2024-04-04T21:02:24.352103+0000","last_became_peered":"2024-04-04T21:02:24.352103+0000","last_unstale":"2024-04-04T21:39:10.982667+0000","last_undegraded":"2024-04-04T21:39:10.982667+0000","last_fullsized":"2024-04-04T21:39:10.982667+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2294,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.245698+0000","last_change":"2024-04-04T21:03:42.104012+0000","last_active":"2024-04-04T21:39:11.245698+0000","last_peered":"2024-04-04T21:39:11.245698+0000","last_clean":"2024-04-04T21:39:11.245698+0000","last_became_active":"2024-04-04T21:03:42.098166+0000","last_became_peered":"2024-04-04T21:03:42.098166+0000","last_unstale":"2024-04-04T21:39:11.245698+0000","last_undegraded":"2024-04-04T21:39:11.245698+0000","last_fullsized":"2024-04-04T21:39:11.245698+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:37:16.291707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2305,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.279018+0000","last_change":"2024-04-04T21:02:24.374143+0000","last_active":"2024-04-04T21:39:11.279018+0000","last_peered":"2024-04-04T21:39:11.279018+0000","last_clean":"2024-04-04T21:39:11.279018+0000","last_became_active":"2024-04-04T21:02:24.359409+0000","last_became_peered":"2024-04-04T21:02:24.359409+0000","last_unstale":"2024-04-04T21:39:11.279018+0000","last_undegraded":"2024-04-04T21:39:11.279018+0000","last_fullsized":"2024-04-04T21:39:11.279018+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:43:28.396677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2318,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:18.718044+0000","last_change":"2024-04-04T21:39:18.718868+0000","last_active":"2024-04-04T21:39:18.718044+0000","last_peered":"2024-04-04T21:39:18.718044+0000","last_clean":"2024-04-04T21:39:18.718044+0000","last_became_active":"2024-04-04T21:03:42.371847+0000","last_became_peered":"2024-04-04T21:03:42.371847+0000","last_unstale":"2024-04-04T21:39:18.718044+0000","last_undegraded":"2024-04-04T21:39:18.718044+0000","last_fullsized":"2024-04-04T21:39:18.718044+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:18.713162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:18.713162+0000","last_clean_scrub_stamp":"2024-04-04T21:39:18.713162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:06:17.152547+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2335,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:19.412329+0000","last_change":"2024-04-04T21:39:19.413259+0000","last_active":"2024-04-04T21:39:19.412329+0000","last_peered":"2024-04-04T21:39:19.412329+0000","last_clean":"2024-04-04T21:39:19.412329+0000","last_became_active":"2024-04-04T21:02:24.236662+0000","last_became_peered":"2024-04-04T21:02:24.236662+0000","last_unstale":"2024-04-04T21:39:19.412329+0000","last_undegraded":"2024-04-04T21:39:19.412329+0000","last_fullsized":"2024-04-04T21:39:19.412329+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:19.407312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:19.407312+0000","last_clean_scrub_stamp":"2024-04-04T21:39:19.407312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T22:28:14.678900+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2312,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.273380+0000","last_change":"2024-04-04T21:03:48.545083+0000","last_active":"2024-04-04T21:39:11.273380+0000","last_peered":"2024-04-04T21:39:11.273380+0000","last_clean":"2024-04-04T21:39:11.273380+0000","last_became_active":"2024-04-04T21:03:48.539598+0000","last_became_peered":"2024-04-04T21:03:48.539598+0000","last_unstale":"2024-04-04T21:39:11.273380+0000","last_undegraded":"2024-04-04T21:39:11.273380+0000","last_fullsized":"2024-04-04T21:39:11.273380+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2327,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:19.198819+0000","last_change":"2024-04-04T21:39:19.199569+0000","last_active":"2024-04-04T21:39:19.198819+0000","last_peered":"2024-04-04T21:39:19.198819+0000","last_clean":"2024-04-04T21:39:19.198819+0000","last_became_active":"2024-04-04T21:02:25.412507+0000","last_became_peered":"2024-04-04T21:02:25.412507+0000","last_unstale":"2024-04-04T21:39:19.198819+0000","last_undegraded":"2024-04-04T21:39:19.198819+0000","last_fullsized":"2024-04-04T21:39:19.198819+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:19.197248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:19.197248+0000","last_clean_scrub_stamp":"2024-04-04T21:39:19.197248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T05:02:02.552596+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2314,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:10.995798+0000","last_change":"2024-04-04T21:03:56.049841+0000","last_active":"2024-04-04T21:39:10.995798+0000","last_peered":"2024-04-04T21:39:10.995798+0000","last_clean":"2024-04-04T21:39:10.995798+0000","last_became_active":"2024-04-04T21:03:56.035576+0000","last_became_peered":"2024-04-04T21:03:56.035576+0000","last_unstale":"2024-04-04T21:39:10.995798+0000","last_undegraded":"2024-04-04T21:39:10.995798+0000","last_fullsized":"2024-04-04T21:39:10.995798+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2327,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:17.721919+0000","last_change":"2024-04-04T21:02:25.403337+0000","last_active":"2024-04-04T21:39:17.721919+0000","last_peered":"2024-04-04T21:39:17.721919+0000","last_clean":"2024-04-04T21:39:17.721919+0000","last_became_active":"2024-04-04T21:02:25.394276+0000","last_became_peered":"2024-04-04T21:02:25.394276+0000","last_unstale":"2024-04-04T21:39:17.721919+0000","last_undegraded":"2024-04-04T21:39:17.721919+0000","last_fullsized":"2024-04-04T21:39:17.721919+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:02:22.675520+0000","last_clean_scrub_stamp":"2024-04-04T21:02:22.675520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2320,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.232673+0000","last_change":"2024-04-04T21:03:55.487132+0000","last_active":"2024-04-04T21:39:11.232673+0000","last_peered":"2024-04-04T21:39:11.232673+0000","last_clean":"2024-04-04T21:39:11.232673+0000","last_became_active":"2024-04-04T21:03:55.479649+0000","last_became_peered":"2024-04-04T21:03:55.479649+0000","last_unstale":"2024-04-04T21:39:11.232673+0000","last_undegraded":"2024-04-04T21:39:11.232673+0000","last_fullsized":"2024-04-04T21:39:11.232673+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:54:34.290339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2341,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:10.934332+0000","last_change":"2024-04-04T21:03:49.740972+0000","last_active":"2024-04-04T21:39:10.934332+0000","last_peered":"2024-04-04T21:39:10.934332+0000","last_clean":"2024-04-04T21:39:10.934332+0000","last_became_active":"2024-04-04T21:03:49.734675+0000","last_became_peered":"2024-04-04T21:03:49.734675+0000","last_unstale":"2024-04-04T21:39:10.934332+0000","last_undegraded":"2024-04-04T21:39:10.934332+0000","last_fullsized":"2024-04-04T21:39:10.934332+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:02:53.025876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2341,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:10.945315+0000","last_change":"2024-04-04T21:03:46.411154+0000","last_active":"2024-04-04T21:39:10.945315+0000","last_peered":"2024-04-04T21:39:10.945315+0000","last_clean":"2024-04-04T21:39:10.945315+0000","last_became_active":"2024-04-04T21:03:46.400875+0000","last_became_peered":"2024-04-04T21:03:46.400875+0000","last_unstale":"2024-04-04T21:39:10.945315+0000","last_undegraded":"2024-04-04T21:39:10.945315+0000","last_fullsized":"2024-04-04T21:39:10.945315+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:55:55.301166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2292,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.276379+0000","last_change":"2024-04-04T21:03:47.549919+0000","last_active":"2024-04-04T21:39:11.276379+0000","last_peered":"2024-04-04T21:39:11.276379+0000","last_clean":"2024-04-04T21:39:11.276379+0000","last_became_active":"2024-04-04T21:03:47.514084+0000","last_became_peered":"2024-04-04T21:03:47.514084+0000","last_unstale":"2024-04-04T21:39:11.276379+0000","last_undegraded":"2024-04-04T21:39:11.276379+0000","last_fullsized":"2024-04-04T21:39:11.276379+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T01:39:02.880998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2340,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:10.940794+0000","last_change":"2024-04-04T21:03:51.865620+0000","last_active":"2024-04-04T21:39:10.940794+0000","last_peered":"2024-04-04T21:39:10.940794+0000","last_clean":"2024-04-04T21:39:10.940794+0000","last_became_active":"2024-04-04T21:03:51.847711+0000","last_became_peered":"2024-04-04T21:03:51.847711+0000","last_unstale":"2024-04-04T21:39:10.940794+0000","last_undegraded":"2024-04-04T21:39:10.940794+0000","last_fullsized":"2024-04-04T21:39:10.940794+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:06:45.768228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2308,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:18.097017+0000","last_change":"2024-04-04T21:39:18.097829+0000","last_active":"2024-04-04T21:39:18.097017+0000","last_peered":"2024-04-04T21:39:18.097017+0000","last_clean":"2024-04-04T21:39:18.097017+0000","last_became_active":"2024-04-04T21:03:40.322843+0000","last_became_peered":"2024-04-04T21:03:40.322843+0000","last_unstale":"2024-04-04T21:39:18.097017+0000","last_undegraded":"2024-04-04T21:39:18.097017+0000","last_fullsized":"2024-04-04T21:39:18.097017+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:18.095566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:18.095566+0000","last_clean_scrub_stamp":"2024-04-04T21:39:18.095566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T23:22:45.482338+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2320,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:17.721649+0000","last_change":"2024-04-04T21:39:17.722466+0000","last_active":"2024-04-04T21:39:17.721649+0000","last_peered":"2024-04-04T21:39:17.721649+0000","last_clean":"2024-04-04T21:39:17.721649+0000","last_became_active":"2024-04-04T21:03:39.803236+0000","last_became_peered":"2024-04-04T21:03:39.803236+0000","last_unstale":"2024-04-04T21:39:17.721649+0000","last_undegraded":"2024-04-04T21:39:17.721649+0000","last_fullsized":"2024-04-04T21:39:17.721649+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:17.719989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:17.719989+0000","last_clean_scrub_stamp":"2024-04-04T21:39:17.719989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T22:39:32.905391+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2308,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:17.120091+0000","last_change":"2024-04-04T21:39:17.120858+0000","last_active":"2024-04-04T21:39:17.120091+0000","last_peered":"2024-04-04T21:39:17.120091+0000","last_clean":"2024-04-04T21:39:17.120091+0000","last_became_active":"2024-04-04T21:03:40.004086+0000","last_became_peered":"2024-04-04T21:03:40.004086+0000","last_unstale":"2024-04-04T21:39:17.120091+0000","last_undegraded":"2024-04-04T21:39:17.120091+0000","last_fullsized":"2024-04-04T21:39:17.120091+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:17.118502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:17.118502+0000","last_clean_scrub_stamp":"2024-04-04T21:39:17.118502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:15:43.597514+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2315,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:10.972983+0000","last_change":"2024-04-04T21:03:43.781555+0000","last_active":"2024-04-04T21:39:10.972983+0000","last_peered":"2024-04-04T21:39:10.972983+0000","last_clean":"2024-04-04T21:39:10.972983+0000","last_became_active":"2024-04-04T21:03:43.766926+0000","last_became_peered":"2024-04-04T21:03:43.766926+0000","last_unstale":"2024-04-04T21:39:10.972983+0000","last_undegraded":"2024-04-04T21:39:10.972983+0000","last_fullsized":"2024-04-04T21:39:10.972983+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2299,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:15.844316+0000","last_change":"2024-04-04T21:39:15.845424+0000","last_active":"2024-04-04T21:39:15.844316+0000","last_peered":"2024-04-04T21:39:15.844316+0000","last_clean":"2024-04-04T21:39:15.844316+0000","last_became_active":"2024-04-04T21:03:43.143344+0000","last_became_peered":"2024-04-04T21:03:43.143344+0000","last_unstale":"2024-04-04T21:39:15.844316+0000","last_undegraded":"2024-04-04T21:39:15.844316+0000","last_fullsized":"2024-04-04T21:39:15.844316+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:15.842688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:15.842688+0000","last_clean_scrub_stamp":"2024-04-04T21:39:15.842688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T23:42:30.212260+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2317,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:19.745461+0000","last_change":"2024-04-04T21:39:19.746220+0000","last_active":"2024-04-04T21:39:19.745461+0000","last_peered":"2024-04-04T21:39:19.745461+0000","last_clean":"2024-04-04T21:39:19.745461+0000","last_became_active":"2024-04-04T21:03:42.379382+0000","last_became_peered":"2024-04-04T21:03:42.379382+0000","last_unstale":"2024-04-04T21:39:19.745461+0000","last_undegraded":"2024-04-04T21:39:19.745461+0000","last_fullsized":"2024-04-04T21:39:19.745461+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:19.743919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:19.743919+0000","last_clean_scrub_stamp":"2024-04-04T21:39:19.743919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T23:25:47.089080+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2293,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:11.273736+0000","last_change":"2024-04-04T21:03:47.554482+0000","last_active":"2024-04-04T21:39:11.273736+0000","last_peered":"2024-04-04T21:39:11.273736+0000","last_clean":"2024-04-04T21:39:11.273736+0000","last_became_active":"2024-04-04T21:03:47.524251+0000","last_became_peered":"2024-04-04T21:03:47.524251+0000","last_unstale":"2024-04-04T21:39:11.273736+0000","last_undegraded":"2024-04-04T21:39:11.273736+0000","last_fullsized":"2024-04-04T21:39:11.273736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:11:26.408285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2313,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:10.993883+0000","last_change":"2024-04-04T21:03:45.814604+0000","last_active":"2024-04-04T21:39:10.993883+0000","last_peered":"2024-04-04T21:39:10.993883+0000","last_clean":"2024-04-04T21:39:10.993883+0000","last_became_active":"2024-04-04T21:03:45.808883+0000","last_became_peered":"2024-04-04T21:03:45.808883+0000","last_unstale":"2024-04-04T21:39:10.993883+0000","last_undegraded":"2024-04-04T21:39:10.993883+0000","last_fullsized":"2024-04-04T21:39:10.993883+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:03:34.739761+0000","last_clean_scrub_stamp":"2024-04-04T21:03:34.739761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899346369,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68352,"kb_used_data":5240,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303488,"statfs":{"total":96636764160,"available":96566771712,"internally_reserved":0,"allocated":5365760,"data_stored":2626920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346369,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76672,"kb_used_data":5240,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295168,"statfs":{"total":96636764160,"available":96558252032,"internally_reserved":0,"allocated":5365760,"data_stored":2626920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346369,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72904,"kb_used_data":5632,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298936,"statfs":{"total":96636764160,"available":96562110464,"internally_reserved":0,"allocated":5767168,"data_stored":3024760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411779,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72516,"kb_used_data":5244,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299324,"statfs":{"total":96636764160,"available":96562507776,"internally_reserved":0,"allocated":5369856,"data_stored":2626925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509891,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72524,"kb_used_data":5240,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299316,"statfs":{"total":96636764160,"available":96562499584,"internally_reserved":0,"allocated":5365760,"data_stored":2626920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509886,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68352,"kb_used_data":5240,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303488,"statfs":{"total":96636764160,"available":96566771712,"internally_reserved":0,"allocated":5365760,"data_stored":2626920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542593,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72520,"kb_used_data":5244,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299320,"statfs":{"total":96636764160,"available":96562503680,"internally_reserved":0,"allocated":5369856,"data_stored":2626925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575295,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68744,"kb_used_data":5632,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303096,"statfs":{"total":96636764160,"available":96566370304,"internally_reserved":0,"allocated":5767168,"data_stored":3024760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-04T21:39:21.817 INFO:tasks.ceph:pgid 9.19 last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.817 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.817 INFO:tasks.ceph:pgid 9.1a last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.817 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.818 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.818 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.818 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.818 INFO:tasks.ceph:pgid 9.1 last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.818 INFO:tasks.ceph:pgid 9.0 last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.818 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.818 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-04T21:02:22.675520+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=2, tm_sec=22, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.818 INFO:tasks.ceph:pgid 9.8 last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.819 INFO:tasks.ceph:pgid 9.b last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.819 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-04T21:02:22.675520+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=2, tm_sec=22, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.819 INFO:tasks.ceph:pgid 9.a last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.819 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-04T21:02:22.675520+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=2, tm_sec=22, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.819 INFO:tasks.ceph:pgid 9.9 last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.819 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-04T21:02:22.675520+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=2, tm_sec=22, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.819 INFO:tasks.ceph:pgid 9.e last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.819 INFO:tasks.ceph:pgid 9.d last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.819 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-04T21:02:22.675520+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=2, tm_sec=22, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.820 INFO:tasks.ceph:pgid 9.c last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.820 INFO:tasks.ceph:pgid 9.1e last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.820 INFO:tasks.ceph:pgid 9.1f last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.820 INFO:tasks.ceph:pgid 9.1c last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.820 INFO:tasks.ceph:pgid 9.1d last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.820 INFO:tasks.ceph:pgid 9.11 last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.820 INFO:tasks.ceph:pgid 9.14 last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.820 INFO:tasks.ceph:pgid 9.15 last_scrub_stamp 2024-04-04T21:03:34.739761+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=95, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=4, tm_hour=21, tm_min=39, tm_sec=13, tm_wday=3, tm_yday=95, tm_isdst=0) 2024-04-04T21:39:21.820 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-04T21:39:26.002 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:17.519 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:26.004 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:17.521 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:26.016 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:17.534 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:26.052 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:17.534 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:26.054 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:17.571 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:26.055 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:17.572 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:26.056 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:17.539 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:26.069 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:17.552 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:26.070 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:17.586 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:41.029 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:32.546 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:41.031 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:32.549 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:41.044 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:32.561 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:41.082 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:32.598 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:41.082 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:32.565 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:41.083 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:32.600 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:41.083 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:32.566 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:41.100 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:32.617 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:41.100 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:32.583 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:41.822 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-04T21:39:41.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.924+0000 7f2b7601b640 10 monclient: get_monmap_and_config 2024-04-04T21:39:41.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.924+0000 7f2b7601b640 10 monclient: build_initial_monmap 2024-04-04T21:39:41.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.924+0000 7f2b7601b640 10 monclient: monmap: 2024-04-04T21:39:41.925 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:41.925 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:41.925 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:41.925891+0000 2024-04-04T21:39:41.925 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:41.925891+0000 2024-04-04T21:39:41.925 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:41.925 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:41.926 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:41.926 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:41.926 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:41.926 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:41.926 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:41.926 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:41.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.925+0000 7f2b7601b640 1 Processor -- start 2024-04-04T21:39:41.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.925+0000 7f2b7601b640 1 -- start start 2024-04-04T21:39:41.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.925+0000 7f2b7601b640 10 monclient: init 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b7601b640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b7601b640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b7601b640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b700a9a50 0x7f2b700a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b7601b640 10 monclient(hunting): picked mon.noname-b con 0x7f2b700a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b7601b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b7013f6a0 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b7601b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2b7013fac0 0x7f2b70142f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b7601b640 10 monclient(hunting): picked mon.noname-c con 0x7f2b7013fac0 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b7601b640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b700ab750 con 0x7f2b7013f6a0 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b7601b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b7601b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b701434d0 con 0x7f2b7013fac0 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b7601b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b7601b640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7013ff50 con 0x7f2b700a9a50 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b7601b640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b7601b640 10 monclient(hunting): authenticate will time out at 3631.193428s 2024-04-04T21:39:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b6f7fe640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b700a9a50 0x7f2b700a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b66ffd640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f2b7013f6a0 0x7f2b70142880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:33990/0 (socket says 172.21.15.106:33990) 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.926+0000 7f2b66ffd640 1 -- 172.21.15.106:0/1208157082 learned_addr learned my addr 172.21.15.106:0/1208157082 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.927+0000 7f2b6ffff640 1 --2- 172.21.15.106:0/1208157082 >> v2:172.21.15.203:3300/0 conn(0x7f2b7013fac0 0x7f2b70142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.927+0000 7f2b6ffff640 10 monclient(hunting): get_auth_request con 0x7f2b7013fac0 auth_method 0 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.927+0000 7f2b6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.927+0000 7f2b6ffff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.927+0000 7f2b6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.927+0000 7f2b6f7fe640 10 monclient(hunting): get_auth_request con 0x7f2b700a9a50 auth_method 0 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.927+0000 7f2b6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.927+0000 7f2b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.927+0000 7f2b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.927+0000 7f2b6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.927+0000 7f2b6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.927+0000 7f2b6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.928+0000 7f2b6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:41.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.928+0000 7f2b6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:41.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.928+0000 7f2b6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:41.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.929+0000 7f2b6effd640 1 -- 172.21.15.106:0/1208157082 <== mon.0 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974769097 0 0) 0x7f2b700ab750 con 0x7f2b7013f6a0 2024-04-04T21:39:41.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.929+0000 7f2b6effd640 10 monclient(hunting): _negotiate 2024-04-04T21:39:41.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.929+0000 7f2b6effd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:41.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.929+0000 7f2b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:41.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.929+0000 7f2b6effd640 10 monclient(hunting): my global_id is 15189 2024-04-04T21:39:41.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.929+0000 7f2b6effd640 1 -- 172.21.15.106:0/1208157082 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b5c003520 con 0x7f2b7013f6a0 2024-04-04T21:39:41.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.930+0000 7f2b6ffff640 10 monclient(hunting): handle_auth_done global_id 22144 payload 274 2024-04-04T21:39:41.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.930+0000 7f2b6ffff640 1 -- 172.21.15.106:0/1208157082 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b700a9a50 msgr2=0x7f2b700a9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:41.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.930+0000 7f2b6ffff640 1 --2- 172.21.15.106:0/1208157082 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b700a9a50 0x7f2b700a9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:41.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.930+0000 7f2b6ffff640 1 -- 172.21.15.106:0/1208157082 >> v1:172.21.15.106:6789/0 conn(0x7f2b7013f6a0 legacy=0x7f2b70142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:41.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.930+0000 7f2b6ffff640 10 monclient: _finish_hunting 0 2024-04-04T21:39:41.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.930+0000 7f2b6ffff640 1 monclient: found mon.noname-c 2024-04-04T21:39:41.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.930+0000 7f2b6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:41.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.930+0000 7f2b6ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-04-04T21:39:41.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.930+0000 7f2b6ffff640 1 -- 172.21.15.106:0/1208157082 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b70140120 con 0x7f2b7013fac0 2024-04-04T21:39:41.934 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.932+0000 7f2b6ffff640 1 --2- 172.21.15.106:0/1208157082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b7013fac0 0x7f2b70142f90 secure :-1 s=READY pgs=5558 cs=0 l=1 rev1=1 crypto rx=0x7f2b68009d90 tx=0x7f2b6802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c656e172ca7e1e88 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:41.936 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b6effd640 1 -- 172.21.15.106:0/1208157082 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b68039070 con 0x7f2b7013fac0 2024-04-04T21:39:41.936 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:41.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b6effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-04T21:39:41.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b6effd640 10 monclient: dump: 2024-04-04T21:39:41.937 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:41.937 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:41.937 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:41.937 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:41.937 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:41.937 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:41.937 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:41.937 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:41.937 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:41.937 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:41.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b6effd640 10 monclient: _finish_auth 0 2024-04-04T21:39:41.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b6effd640 10 monclient: _check_auth_tickets 2024-04-04T21:39:41.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:41.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b6effd640 1 -- 172.21.15.106:0/1208157082 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6802cbc0 con 0x7f2b7013fac0 2024-04-04T21:39:41.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b6effd640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:41.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b6effd640 1 -- 172.21.15.106:0/1208157082 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b68034440 con 0x7f2b7013fac0 2024-04-04T21:39:41.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b7601b640 5 monclient: authenticate success, global_id 22144 2024-04-04T21:39:41.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b7601b640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:41.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:41.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b7601b640 4 set_mon_vals no callback set 2024-04-04T21:39:41.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b6effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:41.938 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.936+0000 7f2b6effd640 10 monclient: dump: 2024-04-04T21:39:41.939 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:41.939 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:41.939 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:41.939 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:41.939 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:41.939 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:41.940 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:41.940 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:41.940 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:41.940 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:41.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.937+0000 7f2b7601b640 10 monclient: shutdown 2024-04-04T21:39:41.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.937+0000 7f2b7601b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:41.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.937+0000 7f2b7601b640 1 -- 172.21.15.106:0/1208157082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b7013fac0 msgr2=0x7f2b70142f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:41.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.937+0000 7f2b7601b640 1 --2- 172.21.15.106:0/1208157082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b7013fac0 0x7f2b70142f90 secure :-1 s=READY pgs=5558 cs=0 l=1 rev1=1 crypto rx=0x7f2b68009d90 tx=0x7f2b6802be10 comp rx=0 tx=0).stop 2024-04-04T21:39:41.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.937+0000 7f2b7601b640 1 -- 172.21.15.106:0/1208157082 shutdown_connections 2024-04-04T21:39:41.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.937+0000 7f2b7601b640 1 --2- 172.21.15.106:0/1208157082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b7013fac0 0x7f2b70142f90 unknown :-1 s=CLOSED pgs=5558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:41.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.937+0000 7f2b7601b640 1 --2- 172.21.15.106:0/1208157082 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b700a9a50 0x7f2b700a9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:41.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.937+0000 7f2b7601b640 1 -- 172.21.15.106:0/1208157082 >> 172.21.15.106:0/1208157082 conn(0x7f2b700b8030 msgr2=0x7f2b700b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:41.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.937+0000 7f2b7601b640 1 -- 172.21.15.106:0/1208157082 shutdown_connections 2024-04-04T21:39:41.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.937+0000 7f2b7601b640 1 -- 172.21.15.106:0/1208157082 wait complete. 2024-04-04T21:39:41.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.938+0000 7f2b7601b640 10 monclient: build_initial_monmap 2024-04-04T21:39:41.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.938+0000 7f2b7601b640 10 monclient: monmap: 2024-04-04T21:39:41.946 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:41.946 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:41.946 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:41.946 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:41.946 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:41.946 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:41.946 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:41.946 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:41.946 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:41.946 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:41.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.938+0000 7f2b7601b640 1 librados: starting msgr at 2024-04-04T21:39:41.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.938+0000 7f2b7601b640 1 librados: starting objecter 2024-04-04T21:39:41.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.938+0000 7f2b7601b640 1 Processor -- start 2024-04-04T21:39:41.946 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 1 -- start start 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 1 librados: setting wanted keys 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 1 librados: calling monclient init 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 10 monclient: init 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b700a9a50 0x7f2b701361f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 10 monclient(hunting): picked mon.noname-a con 0x7f2b700a9a50 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b7013f6a0 0x7f2b701db2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 10 monclient(hunting): picked mon.noname-c con 0x7f2b7013f6a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b7013fac0 0x7f2b701db7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 10 monclient(hunting): picked mon.noname-b con 0x7f2b7013fac0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7014cef0 con 0x7f2b700a9a50 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:41.947 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b700aa370 con 0x7f2b7013fac0 2024-04-04T21:39:41.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:41.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7014d070 con 0x7f2b7013f6a0 2024-04-04T21:39:41.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:41.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.939+0000 7f2b7601b640 10 monclient(hunting): authenticate will time out at 3631.206865s 2024-04-04T21:39:41.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b6f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b700a9a50 0x7f2b701361f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:41.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b66ffd640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b7013f6a0 0x7f2b701db2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:41.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b6ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b7013fac0 0x7f2b701db7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:41.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b6ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b7013fac0 0x7f2b701db7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.106:56126/0 (socket says 172.21.15.106:56126) 2024-04-04T21:39:41.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b66ffd640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b7013f6a0 0x7f2b701db2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:37736/0 (socket says 172.21.15.106:37736) 2024-04-04T21:39:41.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b6ffff640 1 -- 172.21.15.106:0/4117980128 learned_addr learned my addr 172.21.15.106:0/4117980128 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:41.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b6ffff640 10 monclient(hunting): get_auth_request con 0x7f2b7013fac0 auth_method 0 2024-04-04T21:39:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b6ffff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b6f7fe640 10 monclient(hunting): get_auth_request con 0x7f2b700a9a50 auth_method 0 2024-04-04T21:39:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b66ffd640 10 monclient(hunting): get_auth_request con 0x7f2b7013f6a0 auth_method 0 2024-04-04T21:39:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b66ffd640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.940+0000 7f2b66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.941+0000 7f2b6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.941+0000 7f2b6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.941+0000 7f2b6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:41.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.941+0000 7f2b66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:41.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.941+0000 7f2b66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:41.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.941+0000 7f2b66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:41.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.941+0000 7f2b6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:41.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.941+0000 7f2b6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:41.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.941+0000 7f2b6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:41.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.943+0000 7f2b6ffff640 10 monclient(hunting): handle_auth_done global_id 22147 payload 824 2024-04-04T21:39:41.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.943+0000 7f2b6ffff640 1 -- 172.21.15.106:0/4117980128 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b7013f6a0 msgr2=0x7f2b701db2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:41.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.943+0000 7f2b6ffff640 1 --2- 172.21.15.106:0/4117980128 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b7013f6a0 0x7f2b701db2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:41.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.943+0000 7f2b6ffff640 1 -- 172.21.15.106:0/4117980128 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b700a9a50 msgr2=0x7f2b701361f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:41.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.943+0000 7f2b6ffff640 1 --2- 172.21.15.106:0/4117980128 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b700a9a50 0x7f2b701361f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:41.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.943+0000 7f2b6ffff640 10 monclient: _finish_hunting 0 2024-04-04T21:39:41.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.943+0000 7f2b6ffff640 1 monclient: found mon.noname-b 2024-04-04T21:39:41.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.943+0000 7f2b6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:41.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.943+0000 7f2b6ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:41.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.943+0000 7f2b6ffff640 1 -- 172.21.15.106:0/4117980128 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b70137650 con 0x7f2b7013fac0 2024-04-04T21:39:41.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.945+0000 7f2b6ffff640 1 --2- 172.21.15.106:0/4117980128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b7013fac0 0x7f2b701db7f0 secure :-1 s=READY pgs=5559 cs=0 l=1 rev1=1 crypto rx=0x7f2b6802c420 tx=0x7f2b68004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.945+0000 7f2b6cff9640 10 client.?.objecter ms_handle_connect 0x7f2b7013fac0 2024-04-04T21:39:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.945+0000 7f2b6cff9640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b6cff9640 1 -- 172.21.15.106:0/4117980128 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b68044020 con 0x7f2b7013fac0 2024-04-04T21:39:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b6cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b6cff9640 10 monclient: dump: 2024-04-04T21:39:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:41.954 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b6cff9640 10 monclient: _finish_auth 0 2024-04-04T21:39:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b6cff9640 10 monclient: _check_auth_tickets 2024-04-04T21:39:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b6cff9640 1 -- 172.21.15.106:0/4117980128 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b68035e50 con 0x7f2b7013fac0 2024-04-04T21:39:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b7601b640 5 monclient: authenticate success, global_id 22147 2024-04-04T21:39:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b7601b640 10 monclient: _renew_subs 2024-04-04T21:39:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b7601b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b7601b640 1 -- 172.21.15.106:0/4117980128 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b70137930 con 0x7f2b7013fac0 2024-04-04T21:39:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b6cff9640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b6cff9640 1 -- 172.21.15.106:0/4117980128 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b68039040 con 0x7f2b7013fac0 2024-04-04T21:39:41.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:41.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b6cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:41.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b6cff9640 10 monclient: dump: 2024-04-04T21:39:41.955 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:41.955 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:41.955 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:41.955 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:41.955 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:41.956 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:41.956 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:41.956 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:41.956 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:41.956 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:41.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b6e7fc640 4 set_mon_vals no callback set 2024-04-04T21:39:41.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b7601b640 10 client.22147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:41.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b7601b640 10 monclient: _renew_subs 2024-04-04T21:39:41.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b7601b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:41.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b7601b640 1 -- 172.21.15.106:0/4117980128 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b701462c0 con 0x7f2b7013fac0 2024-04-04T21:39:41.956 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.949+0000 7f2b7601b640 1 librados: init done 2024-04-04T21:39:41.957 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.950+0000 7f2b7601b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:41.957 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.950+0000 7f2b7601b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:41.957 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.950+0000 7f2b7601b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:41.957 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.950+0000 7f2b7601b640 1 -- 172.21.15.106:0/4117980128 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b701464c0 con 0x7f2b7013fac0 2024-04-04T21:39:41.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.968+0000 7f2b6cff9640 1 -- 172.21.15.106:0/4117980128 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2b68047020 con 0x7f2b7013fac0 2024-04-04T21:39:41.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.969+0000 7f2b6cff9640 1 --2- 172.21.15.106:0/4117980128 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2b5006ceb0 0x7f2b5006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:41.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.969+0000 7f2b6f7fe640 1 --2- 172.21.15.106:0/4117980128 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2b5006ceb0 0x7f2b5006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:41.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.969+0000 7f2b6f7fe640 10 monclient: get_auth_request con 0x7f2b5006ceb0 auth_method 0 2024-04-04T21:39:41.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.969+0000 7f2b6cff9640 10 client.22147.objecter ms_handle_connect 0x7f2b5006ceb0 2024-04-04T21:39:41.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.969+0000 7f2b6f7fe640 1 --2- 172.21.15.106:0/4117980128 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2b5006ceb0 0x7f2b5006f370 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009b40 tx=0x7f2b60009190 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:41.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.970+0000 7f2b6cff9640 1 -- 172.21.15.106:0/4117980128 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7f2b68079740 con 0x7f2b7013fac0 2024-04-04T21:39:41.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.970+0000 7f2b6cff9640 10 client.22147.objecter ms_dispatch 0x7f2b70000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:41.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.970+0000 7f2b6cff9640 3 client.22147.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:41.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.970+0000 7f2b6cff9640 3 client.22147.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:41.971 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:41.970+0000 7f2b6cff9640 20 client.22147.objecter dump_active .. 0 homeless 2024-04-04T21:39:42.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.209+0000 7f2b6cff9640 1 -- 172.21.15.106:0/4117980128 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2b701464c0 con 0x7f2b7013fac0 2024-04-04T21:39:42.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.209+0000 7f2b6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:42.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.209+0000 7f2b6cff9640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:42.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.426+0000 7f2b7601b640 1 -- 172.21.15.106:0/4117980128 --> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b7014ccd0 con 0x7f2b5006ceb0 2024-04-04T21:39:42.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.429+0000 7f2b6cff9640 1 -- 172.21.15.106:0/4117980128 <== mgr.4117 v2:172.21.15.106:6800/2457463546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+116158 (secure 0 0 0) 0x7f2b7014ccd0 con 0x7f2b5006ceb0 2024-04-04T21:39:42.430 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-04T21:39:42.431 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-04-04T21:39:42.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 10 librados: watch_flush enter 2024-04-04T21:39:42.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 10 librados: watch_flush exit 2024-04-04T21:39:42.433 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 20 client.22147.objecter shutdown clearing up homeless session... 2024-04-04T21:39:42.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 10 client.22147.objecter successfully canceled tick 2024-04-04T21:39:42.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 1 -- 172.21.15.106:0/4117980128 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2b5006ceb0 msgr2=0x7f2b5006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:42.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 1 --2- 172.21.15.106:0/4117980128 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2b5006ceb0 0x7f2b5006f370 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009b40 tx=0x7f2b60009190 comp rx=0 tx=0).stop 2024-04-04T21:39:42.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 10 monclient: shutdown 2024-04-04T21:39:42.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:42.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 1 -- 172.21.15.106:0/4117980128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b7013fac0 msgr2=0x7f2b701db7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:42.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 1 --2- 172.21.15.106:0/4117980128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b7013fac0 0x7f2b701db7f0 secure :-1 s=READY pgs=5559 cs=0 l=1 rev1=1 crypto rx=0x7f2b6802c420 tx=0x7f2b68004a40 comp rx=0 tx=0).stop 2024-04-04T21:39:42.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 1 -- 172.21.15.106:0/4117980128 shutdown_connections 2024-04-04T21:39:42.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 1 --2- 172.21.15.106:0/4117980128 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7f2b5006ceb0 0x7f2b5006f370 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:42.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 1 --2- 172.21.15.106:0/4117980128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b7013fac0 0x7f2b701db7f0 unknown :-1 s=CLOSED pgs=5559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:42.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 1 --2- 172.21.15.106:0/4117980128 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b7013f6a0 0x7f2b701db2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:42.434 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 1 --2- 172.21.15.106:0/4117980128 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b700a9a50 0x7f2b701361f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:42.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.431+0000 7f2b7601b640 1 -- 172.21.15.106:0/4117980128 >> 172.21.15.106:0/4117980128 conn(0x7f2b700b8030 msgr2=0x7f2b700b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:42.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.432+0000 7f2b7601b640 1 -- 172.21.15.106:0/4117980128 shutdown_connections 2024-04-04T21:39:42.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.432+0000 7f2b7601b640 1 -- 172.21.15.106:0/4117980128 wait complete. 2024-04-04T21:39:42.435 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.432+0000 7f2b7601b640 1 librados: shutdown 2024-04-04T21:39:42.443 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":2327,"stamp":"2024-04-04T21:39:41.066110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":572568,"kb_used_data":42712,"kb_used_omap":12,"kb_used_meta":529779,"kb_avail":754402152,"statfs":{"total":773094113280,"available":772507803648,"internally_reserved":0,"allocated":43737088,"data_stored":21811050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":542494257},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002661"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2349,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:25.140062+0000","last_change":"2024-04-04T21:39:25.140830+0000","last_active":"2024-04-04T21:39:25.140062+0000","last_peered":"2024-04-04T21:39:25.140062+0000","last_clean":"2024-04-04T21:39:25.140062+0000","last_became_active":"2024-04-04T21:03:54.933965+0000","last_became_peered":"2024-04-04T21:03:54.933965+0000","last_unstale":"2024-04-04T21:39:25.140062+0000","last_undegraded":"2024-04-04T21:39:25.140062+0000","last_fullsized":"2024-04-04T21:39:25.140062+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:25.138547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:25.138547+0000","last_clean_scrub_stamp":"2024-04-04T21:39:25.138547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:32:02.235246+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2319,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:21.676226+0000","last_change":"2024-04-04T21:39:21.677025+0000","last_active":"2024-04-04T21:39:21.676226+0000","last_peered":"2024-04-04T21:39:21.676226+0000","last_clean":"2024-04-04T21:39:21.676226+0000","last_became_active":"2024-04-04T21:03:44.551107+0000","last_became_peered":"2024-04-04T21:03:44.551107+0000","last_unstale":"2024-04-04T21:39:21.676226+0000","last_undegraded":"2024-04-04T21:39:21.676226+0000","last_fullsized":"2024-04-04T21:39:21.676226+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:21.670236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:21.670236+0000","last_clean_scrub_stamp":"2024-04-04T21:39:21.670236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:12:13.048033+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2315,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:20.196589+0000","last_change":"2024-04-04T21:39:20.197387+0000","last_active":"2024-04-04T21:39:20.196589+0000","last_peered":"2024-04-04T21:39:20.196589+0000","last_clean":"2024-04-04T21:39:20.196589+0000","last_became_active":"2024-04-04T21:03:52.772128+0000","last_became_peered":"2024-04-04T21:03:52.772128+0000","last_unstale":"2024-04-04T21:39:20.196589+0000","last_undegraded":"2024-04-04T21:39:20.196589+0000","last_fullsized":"2024-04-04T21:39:20.196589+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:20.195013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:20.195013+0000","last_clean_scrub_stamp":"2024-04-04T21:39:20.195013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:45:43.100947+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2301,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:22.905844+0000","last_change":"2024-04-04T21:39:22.906615+0000","last_active":"2024-04-04T21:39:22.905844+0000","last_peered":"2024-04-04T21:39:22.905844+0000","last_clean":"2024-04-04T21:39:22.905844+0000","last_became_active":"2024-04-04T21:03:41.346161+0000","last_became_peered":"2024-04-04T21:03:41.346161+0000","last_unstale":"2024-04-04T21:39:22.905844+0000","last_undegraded":"2024-04-04T21:39:22.905844+0000","last_fullsized":"2024-04-04T21:39:22.905844+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:22.904238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:22.904238+0000","last_clean_scrub_stamp":"2024-04-04T21:39:22.904238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:49:00.887979+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2304,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:24.957245+0000","last_change":"2024-04-04T21:39:24.958044+0000","last_active":"2024-04-04T21:39:24.957245+0000","last_peered":"2024-04-04T21:39:24.957245+0000","last_clean":"2024-04-04T21:39:24.957245+0000","last_became_active":"2024-04-04T21:03:42.122180+0000","last_became_peered":"2024-04-04T21:03:42.122180+0000","last_unstale":"2024-04-04T21:39:24.957245+0000","last_undegraded":"2024-04-04T21:39:24.957245+0000","last_fullsized":"2024-04-04T21:39:24.957245+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:24.955748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:24.955748+0000","last_clean_scrub_stamp":"2024-04-04T21:39:24.955748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:43:18.889888+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2331,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:22.471402+0000","last_change":"2024-04-04T21:39:22.472157+0000","last_active":"2024-04-04T21:39:22.471402+0000","last_peered":"2024-04-04T21:39:22.471402+0000","last_clean":"2024-04-04T21:39:22.471402+0000","last_became_active":"2024-04-04T21:03:42.940728+0000","last_became_peered":"2024-04-04T21:03:42.940728+0000","last_unstale":"2024-04-04T21:39:22.471402+0000","last_undegraded":"2024-04-04T21:39:22.471402+0000","last_fullsized":"2024-04-04T21:39:22.471402+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:22.469875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:22.469875+0000","last_clean_scrub_stamp":"2024-04-04T21:39:22.469875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T05:38:32.586809+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2319,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:20.714645+0000","last_change":"2024-04-04T21:39:20.715432+0000","last_active":"2024-04-04T21:39:20.714645+0000","last_peered":"2024-04-04T21:39:20.714645+0000","last_clean":"2024-04-04T21:39:20.714645+0000","last_became_active":"2024-04-04T21:03:43.596620+0000","last_became_peered":"2024-04-04T21:03:43.596620+0000","last_unstale":"2024-04-04T21:39:20.714645+0000","last_undegraded":"2024-04-04T21:39:20.714645+0000","last_fullsized":"2024-04-04T21:39:20.714645+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:20.712950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:20.712950+0000","last_clean_scrub_stamp":"2024-04-04T21:39:20.712950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T09:17:22.759365+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2321,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:23.703179+0000","last_change":"2024-04-04T21:39:23.703946+0000","last_active":"2024-04-04T21:39:23.703179+0000","last_peered":"2024-04-04T21:39:23.703179+0000","last_clean":"2024-04-04T21:39:23.703179+0000","last_became_active":"2024-04-04T21:03:52.351392+0000","last_became_peered":"2024-04-04T21:03:52.351392+0000","last_unstale":"2024-04-04T21:39:23.703179+0000","last_undegraded":"2024-04-04T21:39:23.703179+0000","last_fullsized":"2024-04-04T21:39:23.703179+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:23.701636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:23.701636+0000","last_clean_scrub_stamp":"2024-04-04T21:39:23.701636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T09:38:48.815505+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"1197'8","reported_seq":2346,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:23.536333+0000","last_change":"2024-04-04T21:39:23.537268+0000","last_active":"2024-04-04T21:39:23.536333+0000","last_peered":"2024-04-04T21:39:23.536333+0000","last_clean":"2024-04-04T21:39:23.536333+0000","last_became_active":"2024-04-04T21:03:50.287197+0000","last_became_peered":"2024-04-04T21:03:50.287197+0000","last_unstale":"2024-04-04T21:39:23.536333+0000","last_undegraded":"2024-04-04T21:39:23.536333+0000","last_fullsized":"2024-04-04T21:39:23.536333+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"1197'8","last_scrub_stamp":"2024-04-04T21:39:23.534753+0000","last_deep_scrub":"1197'8","last_deep_scrub_stamp":"2024-04-04T21:39:23.534753+0000","last_clean_scrub_stamp":"2024-04-04T21:39:23.534753+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T22:38:28.814519+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2304,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:25.945537+0000","last_change":"2024-04-04T21:39:25.946288+0000","last_active":"2024-04-04T21:39:25.945537+0000","last_peered":"2024-04-04T21:39:25.945537+0000","last_clean":"2024-04-04T21:39:25.945537+0000","last_became_active":"2024-04-04T21:03:44.913048+0000","last_became_peered":"2024-04-04T21:03:44.913048+0000","last_unstale":"2024-04-04T21:39:25.945537+0000","last_undegraded":"2024-04-04T21:39:25.945537+0000","last_fullsized":"2024-04-04T21:39:25.945537+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:25.943934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:25.943934+0000","last_clean_scrub_stamp":"2024-04-04T21:39:25.943934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T04:58:14.971625+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2302,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:16.850272+0000","last_change":"2024-04-04T21:39:16.851029+0000","last_active":"2024-04-04T21:39:16.850272+0000","last_peered":"2024-04-04T21:39:16.850272+0000","last_clean":"2024-04-04T21:39:16.850272+0000","last_became_active":"2024-04-04T21:03:44.451112+0000","last_became_peered":"2024-04-04T21:03:44.451112+0000","last_unstale":"2024-04-04T21:39:16.850272+0000","last_undegraded":"2024-04-04T21:39:16.850272+0000","last_fullsized":"2024-04-04T21:39:16.850272+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:16.848683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:16.848683+0000","last_clean_scrub_stamp":"2024-04-04T21:39:16.848683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T09:27:15.312906+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2351,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:21.222133+0000","last_change":"2024-04-04T21:39:21.226751+0000","last_active":"2024-04-04T21:39:21.222133+0000","last_peered":"2024-04-04T21:39:21.222133+0000","last_clean":"2024-04-04T21:39:21.222133+0000","last_became_active":"2024-04-04T21:03:44.366452+0000","last_became_peered":"2024-04-04T21:03:44.366452+0000","last_unstale":"2024-04-04T21:39:21.222133+0000","last_undegraded":"2024-04-04T21:39:21.222133+0000","last_fullsized":"2024-04-04T21:39:21.222133+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:21.215189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:21.215189+0000","last_clean_scrub_stamp":"2024-04-04T21:39:21.215189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:09:02.681893+0000","scrub_duration":198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"1197'78","reported_seq":2333,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:16.155555+0000","last_change":"2024-04-04T21:39:16.157056+0000","last_active":"2024-04-04T21:39:16.155555+0000","last_peered":"2024-04-04T21:39:16.155555+0000","last_clean":"2024-04-04T21:39:16.155555+0000","last_became_active":"2024-04-04T21:02:19.876220+0000","last_became_peered":"2024-04-04T21:02:19.876220+0000","last_unstale":"2024-04-04T21:39:16.155555+0000","last_undegraded":"2024-04-04T21:39:16.155555+0000","last_fullsized":"2024-04-04T21:39:16.155555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"1197'78","last_scrub_stamp":"2024-04-04T21:39:16.150979+0000","last_deep_scrub":"1197'78","last_deep_scrub_stamp":"2024-04-04T21:39:16.150979+0000","last_clean_scrub_stamp":"2024-04-04T21:39:16.150979+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T23:01:55.827963+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2336,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:21.466864+0000","last_change":"2024-04-04T21:39:21.468176+0000","last_active":"2024-04-04T21:39:21.466864+0000","last_peered":"2024-04-04T21:39:21.466864+0000","last_clean":"2024-04-04T21:39:21.466864+0000","last_became_active":"2024-04-04T21:02:24.466203+0000","last_became_peered":"2024-04-04T21:02:24.466203+0000","last_unstale":"2024-04-04T21:39:21.466864+0000","last_undegraded":"2024-04-04T21:39:21.466864+0000","last_fullsized":"2024-04-04T21:39:21.466864+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:21.464099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:21.464099+0000","last_clean_scrub_stamp":"2024-04-04T21:39:21.464099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:32:01.735446+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2303,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:21.910548+0000","last_change":"2024-04-04T21:39:21.911349+0000","last_active":"2024-04-04T21:39:21.910548+0000","last_peered":"2024-04-04T21:39:21.910548+0000","last_clean":"2024-04-04T21:39:21.910548+0000","last_became_active":"2024-04-04T21:03:40.758544+0000","last_became_peered":"2024-04-04T21:03:40.758544+0000","last_unstale":"2024-04-04T21:39:21.910548+0000","last_undegraded":"2024-04-04T21:39:21.910548+0000","last_fullsized":"2024-04-04T21:39:21.910548+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:21.909019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:21.909019+0000","last_clean_scrub_stamp":"2024-04-04T21:39:21.909019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T22:56:05.793364+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2337,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:18.382155+0000","last_change":"2024-04-04T21:39:18.382966+0000","last_active":"2024-04-04T21:39:18.382155+0000","last_peered":"2024-04-04T21:39:18.382155+0000","last_clean":"2024-04-04T21:39:18.382155+0000","last_became_active":"2024-04-04T21:02:24.146676+0000","last_became_peered":"2024-04-04T21:02:24.146676+0000","last_unstale":"2024-04-04T21:39:18.382155+0000","last_undegraded":"2024-04-04T21:39:18.382155+0000","last_fullsized":"2024-04-04T21:39:18.382155+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:18.379250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:18.379250+0000","last_clean_scrub_stamp":"2024-04-04T21:39:18.379250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:11:16.619775+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2302,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:20.936801+0000","last_change":"2024-04-04T21:39:20.936801+0000","last_active":"2024-04-04T21:39:20.936801+0000","last_peered":"2024-04-04T21:39:20.936801+0000","last_clean":"2024-04-04T21:39:20.936801+0000","last_became_active":"2024-04-04T21:03:39.382089+0000","last_became_peered":"2024-04-04T21:03:39.382089+0000","last_unstale":"2024-04-04T21:39:20.936801+0000","last_undegraded":"2024-04-04T21:39:20.936801+0000","last_fullsized":"2024-04-04T21:39:20.936801+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:20.934567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:20.934567+0000","last_clean_scrub_stamp":"2024-04-04T21:39:20.934567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T05:57:55.499983+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2343,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:21.448816+0000","last_change":"2024-04-04T21:39:21.449604+0000","last_active":"2024-04-04T21:39:21.448816+0000","last_peered":"2024-04-04T21:39:21.448816+0000","last_clean":"2024-04-04T21:39:21.448816+0000","last_became_active":"2024-04-04T21:02:24.159102+0000","last_became_peered":"2024-04-04T21:02:24.159102+0000","last_unstale":"2024-04-04T21:39:21.448816+0000","last_undegraded":"2024-04-04T21:39:21.448816+0000","last_fullsized":"2024-04-04T21:39:21.448816+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:21.447134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:21.447134+0000","last_clean_scrub_stamp":"2024-04-04T21:39:21.447134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T09:12:05.935164+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2309,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:19.125380+0000","last_change":"2024-04-04T21:39:19.126120+0000","last_active":"2024-04-04T21:39:19.125380+0000","last_peered":"2024-04-04T21:39:19.125380+0000","last_clean":"2024-04-04T21:39:19.125380+0000","last_became_active":"2024-04-04T21:03:45.238093+0000","last_became_peered":"2024-04-04T21:03:45.238093+0000","last_unstale":"2024-04-04T21:39:19.125380+0000","last_undegraded":"2024-04-04T21:39:19.125380+0000","last_fullsized":"2024-04-04T21:39:19.125380+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:19.123887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:19.123887+0000","last_clean_scrub_stamp":"2024-04-04T21:39:19.123887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:07:17.576137+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2336,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:20.419366+0000","last_change":"2024-04-04T21:39:20.420179+0000","last_active":"2024-04-04T21:39:20.419366+0000","last_peered":"2024-04-04T21:39:20.419366+0000","last_clean":"2024-04-04T21:39:20.419366+0000","last_became_active":"2024-04-04T21:02:24.352103+0000","last_became_peered":"2024-04-04T21:02:24.352103+0000","last_unstale":"2024-04-04T21:39:20.419366+0000","last_undegraded":"2024-04-04T21:39:20.419366+0000","last_fullsized":"2024-04-04T21:39:20.419366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:20.417780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:20.417780+0000","last_clean_scrub_stamp":"2024-04-04T21:39:20.417780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T01:29:33.888244+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2303,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:23.923288+0000","last_change":"2024-04-04T21:39:23.924046+0000","last_active":"2024-04-04T21:39:23.923288+0000","last_peered":"2024-04-04T21:39:23.923288+0000","last_clean":"2024-04-04T21:39:23.923288+0000","last_became_active":"2024-04-04T21:03:42.098166+0000","last_became_peered":"2024-04-04T21:03:42.098166+0000","last_unstale":"2024-04-04T21:39:23.923288+0000","last_undegraded":"2024-04-04T21:39:23.923288+0000","last_fullsized":"2024-04-04T21:39:23.923288+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:23.921756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:23.921756+0000","last_clean_scrub_stamp":"2024-04-04T21:39:23.921756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T01:17:19.287071+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2314,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:19.849856+0000","last_change":"2024-04-04T21:39:19.850618+0000","last_active":"2024-04-04T21:39:19.849856+0000","last_peered":"2024-04-04T21:39:19.849856+0000","last_clean":"2024-04-04T21:39:19.849856+0000","last_became_active":"2024-04-04T21:02:24.359409+0000","last_became_peered":"2024-04-04T21:02:24.359409+0000","last_unstale":"2024-04-04T21:39:19.849856+0000","last_undegraded":"2024-04-04T21:39:19.849856+0000","last_fullsized":"2024-04-04T21:39:19.849856+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:19.848231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:19.848231+0000","last_clean_scrub_stamp":"2024-04-04T21:39:19.848231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T06:10:02.491996+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2318,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:18.718044+0000","last_change":"2024-04-04T21:39:18.718868+0000","last_active":"2024-04-04T21:39:18.718044+0000","last_peered":"2024-04-04T21:39:18.718044+0000","last_clean":"2024-04-04T21:39:18.718044+0000","last_became_active":"2024-04-04T21:03:42.371847+0000","last_became_peered":"2024-04-04T21:03:42.371847+0000","last_unstale":"2024-04-04T21:39:18.718044+0000","last_undegraded":"2024-04-04T21:39:18.718044+0000","last_fullsized":"2024-04-04T21:39:18.718044+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:18.713162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:18.713162+0000","last_clean_scrub_stamp":"2024-04-04T21:39:18.713162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:06:17.152547+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2335,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:19.412329+0000","last_change":"2024-04-04T21:39:19.413259+0000","last_active":"2024-04-04T21:39:19.412329+0000","last_peered":"2024-04-04T21:39:19.412329+0000","last_clean":"2024-04-04T21:39:19.412329+0000","last_became_active":"2024-04-04T21:02:24.236662+0000","last_became_peered":"2024-04-04T21:02:24.236662+0000","last_unstale":"2024-04-04T21:39:19.412329+0000","last_undegraded":"2024-04-04T21:39:19.412329+0000","last_fullsized":"2024-04-04T21:39:19.412329+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:19.407312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:19.407312+0000","last_clean_scrub_stamp":"2024-04-04T21:39:19.407312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T22:28:14.678900+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2320,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:22.654559+0000","last_change":"2024-04-04T21:39:22.655356+0000","last_active":"2024-04-04T21:39:22.654559+0000","last_peered":"2024-04-04T21:39:22.654559+0000","last_clean":"2024-04-04T21:39:22.654559+0000","last_became_active":"2024-04-04T21:03:48.539598+0000","last_became_peered":"2024-04-04T21:03:48.539598+0000","last_unstale":"2024-04-04T21:39:22.654559+0000","last_undegraded":"2024-04-04T21:39:22.654559+0000","last_fullsized":"2024-04-04T21:39:22.654559+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:22.652946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:22.652946+0000","last_clean_scrub_stamp":"2024-04-04T21:39:22.652946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:04:32.712651+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2327,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:19.198819+0000","last_change":"2024-04-04T21:39:19.199569+0000","last_active":"2024-04-04T21:39:19.198819+0000","last_peered":"2024-04-04T21:39:19.198819+0000","last_clean":"2024-04-04T21:39:19.198819+0000","last_became_active":"2024-04-04T21:02:25.412507+0000","last_became_peered":"2024-04-04T21:02:25.412507+0000","last_unstale":"2024-04-04T21:39:19.198819+0000","last_undegraded":"2024-04-04T21:39:19.198819+0000","last_fullsized":"2024-04-04T21:39:19.198819+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:19.197248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:19.197248+0000","last_clean_scrub_stamp":"2024-04-04T21:39:19.197248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T05:02:02.552596+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2322,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:25.525527+0000","last_change":"2024-04-04T21:39:25.526367+0000","last_active":"2024-04-04T21:39:25.525527+0000","last_peered":"2024-04-04T21:39:25.525527+0000","last_clean":"2024-04-04T21:39:25.525527+0000","last_became_active":"2024-04-04T21:03:56.035576+0000","last_became_peered":"2024-04-04T21:03:56.035576+0000","last_unstale":"2024-04-04T21:39:25.525527+0000","last_undegraded":"2024-04-04T21:39:25.525527+0000","last_fullsized":"2024-04-04T21:39:25.525527+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:25.524013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:25.524013+0000","last_clean_scrub_stamp":"2024-04-04T21:39:25.524013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T09:34:00.773933+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2335,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:22.495116+0000","last_change":"2024-04-04T21:39:22.495853+0000","last_active":"2024-04-04T21:39:22.495116+0000","last_peered":"2024-04-04T21:39:22.495116+0000","last_clean":"2024-04-04T21:39:22.495116+0000","last_became_active":"2024-04-04T21:02:25.394276+0000","last_became_peered":"2024-04-04T21:02:25.394276+0000","last_unstale":"2024-04-04T21:39:22.495116+0000","last_undegraded":"2024-04-04T21:39:22.495116+0000","last_fullsized":"2024-04-04T21:39:22.495116+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:22.493657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:22.493657+0000","last_clean_scrub_stamp":"2024-04-04T21:39:22.493657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T06:08:59.288614+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2329,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:24.423376+0000","last_change":"2024-04-04T21:39:24.424177+0000","last_active":"2024-04-04T21:39:24.423376+0000","last_peered":"2024-04-04T21:39:24.423376+0000","last_clean":"2024-04-04T21:39:24.423376+0000","last_became_active":"2024-04-04T21:03:55.479649+0000","last_became_peered":"2024-04-04T21:03:55.479649+0000","last_unstale":"2024-04-04T21:39:24.423376+0000","last_undegraded":"2024-04-04T21:39:24.423376+0000","last_fullsized":"2024-04-04T21:39:24.423376+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:24.421813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:24.421813+0000","last_clean_scrub_stamp":"2024-04-04T21:39:24.421813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:07:54.215854+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2351,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:23.070654+0000","last_change":"2024-04-04T21:39:23.071457+0000","last_active":"2024-04-04T21:39:23.070654+0000","last_peered":"2024-04-04T21:39:23.070654+0000","last_clean":"2024-04-04T21:39:23.070654+0000","last_became_active":"2024-04-04T21:03:49.734675+0000","last_became_peered":"2024-04-04T21:03:49.734675+0000","last_unstale":"2024-04-04T21:39:23.070654+0000","last_undegraded":"2024-04-04T21:39:23.070654+0000","last_fullsized":"2024-04-04T21:39:23.070654+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:23.069139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:23.069139+0000","last_clean_scrub_stamp":"2024-04-04T21:39:23.069139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T02:04:20.587384+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2350,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:22.066225+0000","last_change":"2024-04-04T21:39:22.067051+0000","last_active":"2024-04-04T21:39:22.066225+0000","last_peered":"2024-04-04T21:39:22.066225+0000","last_clean":"2024-04-04T21:39:22.066225+0000","last_became_active":"2024-04-04T21:03:46.400875+0000","last_became_peered":"2024-04-04T21:03:46.400875+0000","last_unstale":"2024-04-04T21:39:22.066225+0000","last_undegraded":"2024-04-04T21:39:22.066225+0000","last_fullsized":"2024-04-04T21:39:22.066225+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:22.064595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:22.064595+0000","last_clean_scrub_stamp":"2024-04-04T21:39:22.064595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T00:54:28.330737+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2301,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:26.958406+0000","last_change":"2024-04-04T21:39:26.959160+0000","last_active":"2024-04-04T21:39:26.958406+0000","last_peered":"2024-04-04T21:39:26.958406+0000","last_clean":"2024-04-04T21:39:26.958406+0000","last_became_active":"2024-04-04T21:03:47.514084+0000","last_became_peered":"2024-04-04T21:03:47.514084+0000","last_unstale":"2024-04-04T21:39:26.958406+0000","last_undegraded":"2024-04-04T21:39:26.958406+0000","last_fullsized":"2024-04-04T21:39:26.958406+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:26.956881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:26.956881+0000","last_clean_scrub_stamp":"2024-04-04T21:39:26.956881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T06:02:42.045539+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2349,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:24.135746+0000","last_change":"2024-04-04T21:39:24.136742+0000","last_active":"2024-04-04T21:39:24.135746+0000","last_peered":"2024-04-04T21:39:24.135746+0000","last_clean":"2024-04-04T21:39:24.135746+0000","last_became_active":"2024-04-04T21:03:51.847711+0000","last_became_peered":"2024-04-04T21:03:51.847711+0000","last_unstale":"2024-04-04T21:39:24.135746+0000","last_undegraded":"2024-04-04T21:39:24.135746+0000","last_fullsized":"2024-04-04T21:39:24.135746+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:24.134182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:24.134182+0000","last_clean_scrub_stamp":"2024-04-04T21:39:24.134182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:34:39.479282+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2308,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:18.097017+0000","last_change":"2024-04-04T21:39:18.097829+0000","last_active":"2024-04-04T21:39:18.097017+0000","last_peered":"2024-04-04T21:39:18.097017+0000","last_clean":"2024-04-04T21:39:18.097017+0000","last_became_active":"2024-04-04T21:03:40.322843+0000","last_became_peered":"2024-04-04T21:03:40.322843+0000","last_unstale":"2024-04-04T21:39:18.097017+0000","last_undegraded":"2024-04-04T21:39:18.097017+0000","last_fullsized":"2024-04-04T21:39:18.097017+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:18.095566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:18.095566+0000","last_clean_scrub_stamp":"2024-04-04T21:39:18.095566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T23:22:45.482338+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2320,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:17.721649+0000","last_change":"2024-04-04T21:39:17.722466+0000","last_active":"2024-04-04T21:39:17.721649+0000","last_peered":"2024-04-04T21:39:17.721649+0000","last_clean":"2024-04-04T21:39:17.721649+0000","last_became_active":"2024-04-04T21:03:39.803236+0000","last_became_peered":"2024-04-04T21:03:39.803236+0000","last_unstale":"2024-04-04T21:39:17.721649+0000","last_undegraded":"2024-04-04T21:39:17.721649+0000","last_fullsized":"2024-04-04T21:39:17.721649+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:17.719989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:17.719989+0000","last_clean_scrub_stamp":"2024-04-04T21:39:17.719989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T22:39:32.905391+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2308,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:17.120091+0000","last_change":"2024-04-04T21:39:17.120858+0000","last_active":"2024-04-04T21:39:17.120091+0000","last_peered":"2024-04-04T21:39:17.120091+0000","last_clean":"2024-04-04T21:39:17.120091+0000","last_became_active":"2024-04-04T21:03:40.004086+0000","last_became_peered":"2024-04-04T21:03:40.004086+0000","last_unstale":"2024-04-04T21:39:17.120091+0000","last_undegraded":"2024-04-04T21:39:17.120091+0000","last_fullsized":"2024-04-04T21:39:17.120091+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:17.118502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:17.118502+0000","last_clean_scrub_stamp":"2024-04-04T21:39:17.118502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T08:15:43.597514+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2323,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:23.528694+0000","last_change":"2024-04-04T21:39:23.529444+0000","last_active":"2024-04-04T21:39:23.528694+0000","last_peered":"2024-04-04T21:39:23.528694+0000","last_clean":"2024-04-04T21:39:23.528694+0000","last_became_active":"2024-04-04T21:03:43.766926+0000","last_became_peered":"2024-04-04T21:03:43.766926+0000","last_unstale":"2024-04-04T21:39:23.528694+0000","last_undegraded":"2024-04-04T21:39:23.528694+0000","last_fullsized":"2024-04-04T21:39:23.528694+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:23.527231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:23.527231+0000","last_clean_scrub_stamp":"2024-04-04T21:39:23.527231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T05:17:56.836972+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2299,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:15.844316+0000","last_change":"2024-04-04T21:39:15.845424+0000","last_active":"2024-04-04T21:39:15.844316+0000","last_peered":"2024-04-04T21:39:15.844316+0000","last_clean":"2024-04-04T21:39:15.844316+0000","last_became_active":"2024-04-04T21:03:43.143344+0000","last_became_peered":"2024-04-04T21:03:43.143344+0000","last_unstale":"2024-04-04T21:39:15.844316+0000","last_undegraded":"2024-04-04T21:39:15.844316+0000","last_fullsized":"2024-04-04T21:39:15.844316+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:15.842688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:15.842688+0000","last_clean_scrub_stamp":"2024-04-04T21:39:15.842688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T23:42:30.212260+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2317,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:19.745461+0000","last_change":"2024-04-04T21:39:19.746220+0000","last_active":"2024-04-04T21:39:19.745461+0000","last_peered":"2024-04-04T21:39:19.745461+0000","last_clean":"2024-04-04T21:39:19.745461+0000","last_became_active":"2024-04-04T21:03:42.379382+0000","last_became_peered":"2024-04-04T21:03:42.379382+0000","last_unstale":"2024-04-04T21:39:19.745461+0000","last_undegraded":"2024-04-04T21:39:19.745461+0000","last_fullsized":"2024-04-04T21:39:19.745461+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:19.743919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:19.743919+0000","last_clean_scrub_stamp":"2024-04-04T21:39:19.743919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-05T23:25:47.089080+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2302,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:27.917875+0000","last_change":"2024-04-04T21:39:27.918678+0000","last_active":"2024-04-04T21:39:27.917875+0000","last_peered":"2024-04-04T21:39:27.917875+0000","last_clean":"2024-04-04T21:39:27.917875+0000","last_became_active":"2024-04-04T21:03:47.524251+0000","last_became_peered":"2024-04-04T21:03:47.524251+0000","last_unstale":"2024-04-04T21:39:27.917875+0000","last_undegraded":"2024-04-04T21:39:27.917875+0000","last_fullsized":"2024-04-04T21:39:27.917875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:27.913774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:27.913774+0000","last_clean_scrub_stamp":"2024-04-04T21:39:27.913774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T05:03:29.029916+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2321,"reported_epoch":1197,"state":"active+clean","last_fresh":"2024-04-04T21:39:24.547134+0000","last_change":"2024-04-04T21:39:24.547870+0000","last_active":"2024-04-04T21:39:24.547134+0000","last_peered":"2024-04-04T21:39:24.547134+0000","last_clean":"2024-04-04T21:39:24.547134+0000","last_became_active":"2024-04-04T21:03:45.808883+0000","last_became_peered":"2024-04-04T21:03:45.808883+0000","last_unstale":"2024-04-04T21:39:24.547134+0000","last_undegraded":"2024-04-04T21:39:24.547134+0000","last_fullsized":"2024-04-04T21:39:24.547134+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-04T21:39:24.545659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-04T21:39:24.545659+0000","last_clean_scrub_stamp":"2024-04-04T21:39:24.545659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T07:36:01.660100+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899346373,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68352,"kb_used_data":5240,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303488,"statfs":{"total":96636764160,"available":96566771712,"internally_reserved":0,"allocated":5365760,"data_stored":2626920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76672,"kb_used_data":5240,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295168,"statfs":{"total":96636764160,"available":96558252032,"internally_reserved":0,"allocated":5365760,"data_stored":2626920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72904,"kb_used_data":5632,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298936,"statfs":{"total":96636764160,"available":96562110464,"internally_reserved":0,"allocated":5767168,"data_stored":3024760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411783,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72516,"kb_used_data":5244,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299324,"statfs":{"total":96636764160,"available":96562507776,"internally_reserved":0,"allocated":5369856,"data_stored":2626925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509895,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72512,"kb_used_data":5240,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299328,"statfs":{"total":96636764160,"available":96562511872,"internally_reserved":0,"allocated":5365760,"data_stored":2626920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509890,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68352,"kb_used_data":5240,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303488,"statfs":{"total":96636764160,"available":96566771712,"internally_reserved":0,"allocated":5365760,"data_stored":2626920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542597,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72516,"kb_used_data":5244,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299324,"statfs":{"total":96636764160,"available":96562507776,"internally_reserved":0,"allocated":5369856,"data_stored":2626925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575299,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68744,"kb_used_data":5632,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303096,"statfs":{"total":96636764160,"available":96566370304,"internally_reserved":0,"allocated":5767168,"data_stored":3024760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-04T21:39:42.447 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-04T21:39:42.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.550+0000 7fb3f7185640 10 monclient: get_monmap_and_config 2024-04-04T21:39:42.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.550+0000 7fb3f7185640 10 monclient: build_initial_monmap 2024-04-04T21:39:42.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.550+0000 7fb3f7185640 10 monclient: monmap: 2024-04-04T21:39:42.551 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:42.551 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:42.551 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:39:42.551190+0000 2024-04-04T21:39:42.551 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:39:42.551190+0000 2024-04-04T21:39:42.551 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:42.551 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:42.551 INFO:teuthology.orchestra.run.smithi106.stderr:0: v1:172.21.15.106:6789/0 mon.noname-a-legacy 2024-04-04T21:39:42.551 INFO:teuthology.orchestra.run.smithi106.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-04-04T21:39:42.551 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-b 2024-04-04T21:39:42.551 INFO:teuthology.orchestra.run.smithi106.stderr:3: v2:172.21.15.106:3300/0 mon.noname-a 2024-04-04T21:39:42.552 INFO:teuthology.orchestra.run.smithi106.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-04-04T21:39:42.552 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:42.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.550+0000 7fb3f7185640 1 Processor -- start 2024-04-04T21:39:42.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.550+0000 7fb3f7185640 1 -- start start 2024-04-04T21:39:42.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.550+0000 7fb3f7185640 10 monclient: init 2024-04-04T21:39:42.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.551+0000 7fb3f7185640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:42.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.551+0000 7fb3f7185640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-04-04T21:39:42.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.551+0000 7fb3f7185640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3f00a9a50 0x7fb3f00a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:42.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.551+0000 7fb3f7185640 10 monclient(hunting): picked mon.noname-b con 0x7fb3f00a9a50 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:42.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.551+0000 7fb3f7185640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3f013f6a0 addr v1:172.21.15.106:6789/0 2024-04-04T21:39:42.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.551+0000 7fb3f7185640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb3f013fac0 0x7fb3f0142f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:42.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.551+0000 7fb3f7185640 10 monclient(hunting): picked mon.noname-c con 0x7fb3f013fac0 addr v2:172.21.15.203:3300/0 2024-04-04T21:39:42.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.551+0000 7fb3f7185640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f00ab750 con 0x7fb3f013f6a0 2024-04-04T21:39:42.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.551+0000 7fb3f7185640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:42.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.551+0000 7fb3f7185640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f01434d0 con 0x7fb3f013fac0 2024-04-04T21:39:42.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.551+0000 7fb3f7185640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:42.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.551+0000 7fb3f7185640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f013ff50 con 0x7fb3f00a9a50 2024-04-04T21:39:42.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.551+0000 7fb3f7185640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:42.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.551+0000 7fb3f7185640 10 monclient(hunting): authenticate will time out at 3631.818678s 2024-04-04T21:39:42.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.552+0000 7fb3f4f21640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3f00a9a50 0x7fb3f00a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:42.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.552+0000 7fb3effff640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fb3f013f6a0 0x7fb3f0142880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:33994/0 (socket says 172.21.15.106:33994) 2024-04-04T21:39:42.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.552+0000 7fb3effff640 1 -- 172.21.15.106:0/797740556 learned_addr learned my addr 172.21.15.106:0/797740556 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-04-04T21:39:42.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.552+0000 7fb3f5722640 1 --2- 172.21.15.106:0/797740556 >> v2:172.21.15.203:3300/0 conn(0x7fb3f013fac0 0x7fb3f0142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:42.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.552+0000 7fb3f4f21640 10 monclient(hunting): get_auth_request con 0x7fb3f00a9a50 auth_method 0 2024-04-04T21:39:42.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.552+0000 7fb3f4f21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:42.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.552+0000 7fb3f4f21640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:42.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.552+0000 7fb3f4f21640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:42.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.552+0000 7fb3f5722640 10 monclient(hunting): get_auth_request con 0x7fb3f013fac0 auth_method 0 2024-04-04T21:39:42.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.552+0000 7fb3f5722640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:42.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.552+0000 7fb3f5722640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:42.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.552+0000 7fb3f5722640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.553+0000 7fb3f4f21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.553+0000 7fb3f4f21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.553+0000 7fb3f4f21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.553+0000 7fb3f5722640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.553+0000 7fb3f5722640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.553+0000 7fb3f5722640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.554+0000 7fb3f4f21640 10 monclient(hunting): handle_auth_done global_id 17720 payload 274 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.554+0000 7fb3f4f21640 1 -- 172.21.15.106:0/797740556 >> v2:172.21.15.203:3300/0 conn(0x7fb3f013fac0 msgr2=0x7fb3f0142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.554+0000 7fb3f4f21640 1 --2- 172.21.15.106:0/797740556 >> v2:172.21.15.203:3300/0 conn(0x7fb3f013fac0 0x7fb3f0142f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.554+0000 7fb3f4f21640 1 -- 172.21.15.106:0/797740556 >> v1:172.21.15.106:6789/0 conn(0x7fb3f013f6a0 legacy=0x7fb3f0142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.554+0000 7fb3f4f21640 10 monclient: _finish_hunting 0 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.554+0000 7fb3f4f21640 1 monclient: found mon.noname-b 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.554+0000 7fb3f4f21640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.554+0000 7fb3f4f21640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.106:3301/0 2024-04-04T21:39:42.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.554+0000 7fb3f4f21640 1 -- 172.21.15.106:0/797740556 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f0140120 con 0x7fb3f00a9a50 2024-04-04T21:39:42.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.556+0000 7fb3f4f21640 1 --2- 172.21.15.106:0/797740556 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3f00a9a50 0x7fb3f00a9e30 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7fb3e00097a0 tx=0x7fb3e002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8419e2656e93b6a server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:42.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3ef7fe640 1 -- 172.21.15.106:0/797740556 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e00315a0 con 0x7fb3f00a9a50 2024-04-04T21:39:42.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:42.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:42.560 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3ef7fe640 10 monclient: dump: 2024-04-04T21:39:42.560 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:42.560 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:42.560 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:42.560 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:42.560 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:42.560 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:42.560 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:42.560 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:42.560 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:42.560 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:42.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3ef7fe640 10 monclient: _finish_auth 0 2024-04-04T21:39:42.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3ef7fe640 10 monclient: _check_auth_tickets 2024-04-04T21:39:42.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:42.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3ef7fe640 1 -- 172.21.15.106:0/797740556 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e0037070 con 0x7fb3f00a9a50 2024-04-04T21:39:42.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:42.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3ef7fe640 1 -- 172.21.15.106:0/797740556 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0031960 con 0x7fb3f00a9a50 2024-04-04T21:39:42.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3f7185640 5 monclient: authenticate success, global_id 17720 2024-04-04T21:39:42.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3f7185640 10 monclient: get_monmap_and_config success 2024-04-04T21:39:42.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:42.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3f7185640 4 set_mon_vals no callback set 2024-04-04T21:39:42.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-04T21:39:42.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.559+0000 7fb3ef7fe640 10 monclient: dump: 2024-04-04T21:39:42.561 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:42.561 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:42.562 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:42.562 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:42.562 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:42.562 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:42.562 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:42.562 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:42.562 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:42.562 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:42.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.560+0000 7fb3f7185640 10 monclient: shutdown 2024-04-04T21:39:42.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.560+0000 7fb3f7185640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:42.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.560+0000 7fb3f7185640 1 -- 172.21.15.106:0/797740556 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3f00a9a50 msgr2=0x7fb3f00a9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:42.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.560+0000 7fb3f7185640 1 --2- 172.21.15.106:0/797740556 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3f00a9a50 0x7fb3f00a9e30 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7fb3e00097a0 tx=0x7fb3e002b840 comp rx=0 tx=0).stop 2024-04-04T21:39:42.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.560+0000 7fb3f7185640 1 -- 172.21.15.106:0/797740556 shutdown_connections 2024-04-04T21:39:42.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.560+0000 7fb3f7185640 1 --2- 172.21.15.106:0/797740556 >> v2:172.21.15.203:3300/0 conn(0x7fb3f013fac0 0x7fb3f0142f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:42.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.560+0000 7fb3f7185640 1 --2- 172.21.15.106:0/797740556 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3f00a9a50 0x7fb3f00a9e30 unknown :-1 s=CLOSED pgs=3275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:42.566 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.560+0000 7fb3f7185640 1 -- 172.21.15.106:0/797740556 >> 172.21.15.106:0/797740556 conn(0x7fb3f00b8030 msgr2=0x7fb3f00b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:42.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.560+0000 7fb3f7185640 1 -- 172.21.15.106:0/797740556 shutdown_connections 2024-04-04T21:39:42.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.560+0000 7fb3f7185640 1 -- 172.21.15.106:0/797740556 wait complete. 2024-04-04T21:39:42.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.561+0000 7fb3f7185640 10 monclient: build_initial_monmap 2024-04-04T21:39:42.567 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.561+0000 7fb3f7185640 10 monclient: monmap: 2024-04-04T21:39:42.567 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 0 2024-04-04T21:39:42.567 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-04T21:39:42.567 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 0.000000 2024-04-04T21:39:42.567 INFO:teuthology.orchestra.run.smithi106.stderr:created 0.000000 2024-04-04T21:39:42.567 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 0 (unknown) 2024-04-04T21:39:42.567 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:42.567 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.noname-a 2024-04-04T21:39:42.567 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-04-04T21:39:42.567 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.noname-c 2024-04-04T21:39:42.567 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.561+0000 7fb3f7185640 1 librados: starting msgr at 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.561+0000 7fb3f7185640 1 librados: starting objecter 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.561+0000 7fb3f7185640 1 Processor -- start 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.561+0000 7fb3f7185640 1 -- start start 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.561+0000 7fb3f7185640 1 librados: setting wanted keys 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.561+0000 7fb3f7185640 1 librados: calling monclient init 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.561+0000 7fb3f7185640 10 monclient: init 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 10 monclient: _reopen_session rank -1 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3f00a9a50 0x7fb3f0146460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 10 monclient(hunting): picked mon.noname-b con 0x7fb3f00a9a50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3f013f6a0 0x7fb3f01dbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 10 monclient(hunting): picked mon.noname-c con 0x7fb3f013f6a0 addr [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3f013fac0 0x7fb3f013a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 10 monclient(hunting): picked mon.noname-a con 0x7fb3f013fac0 addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-04-04T21:39:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f014cef0 con 0x7fb3f013fac0 2024-04-04T21:39:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f00aa370 con 0x7fb3f00a9a50 2024-04-04T21:39:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 10 monclient(hunting): start opening mon connection 2024-04-04T21:39:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f014d070 con 0x7fb3f013f6a0 2024-04-04T21:39:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 10 monclient(hunting): _renew_subs 2024-04-04T21:39:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f7185640 10 monclient(hunting): authenticate will time out at 3631.829642s 2024-04-04T21:39:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3effff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3f013f6a0 0x7fb3f01dbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.562+0000 7fb3f5722640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3f013fac0 0x7fb3f013a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3f4f21640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3f00a9a50 0x7fb3f0146460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3effff640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3f013f6a0 0x7fb3f01dbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:37762/0 (socket says 172.21.15.106:37762) 2024-04-04T21:39:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3effff640 1 -- 172.21.15.106:0/3624816212 learned_addr learned my addr 172.21.15.106:0/3624816212 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-04-04T21:39:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3effff640 10 monclient(hunting): get_auth_request con 0x7fb3f013f6a0 auth_method 0 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3effff640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3effff640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3f4f21640 10 monclient(hunting): get_auth_request con 0x7fb3f00a9a50 auth_method 0 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3f4f21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3f4f21640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3f4f21640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3f5722640 10 monclient(hunting): get_auth_request con 0x7fb3f013fac0 auth_method 0 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3f5722640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3f5722640 10 monclient(hunting): _init_auth method 2 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3f5722640 10 monclient(hunting): _init_auth creating new auth 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:42.570 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3f4f21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:42.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3f4f21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:42.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.563+0000 7fb3f4f21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:42.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.564+0000 7fb3f5722640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-04T21:39:42.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.564+0000 7fb3f5722640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-04T21:39:42.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.564+0000 7fb3f5722640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-04T21:39:42.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.566+0000 7fb3f4f21640 10 monclient(hunting): handle_auth_done global_id 22153 payload 824 2024-04-04T21:39:42.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.566+0000 7fb3f4f21640 1 -- 172.21.15.106:0/3624816212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3f013f6a0 msgr2=0x7fb3f01dbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:42.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.566+0000 7fb3f4f21640 1 --2- 172.21.15.106:0/3624816212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3f013f6a0 0x7fb3f01dbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:42.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.566+0000 7fb3f4f21640 1 -- 172.21.15.106:0/3624816212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3f013fac0 msgr2=0x7fb3f013a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:42.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.566+0000 7fb3f4f21640 1 --2- 172.21.15.106:0/3624816212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3f013fac0 0x7fb3f013a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:42.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.566+0000 7fb3f4f21640 10 monclient: _finish_hunting 0 2024-04-04T21:39:42.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.566+0000 7fb3f4f21640 1 monclient: found mon.noname-b 2024-04-04T21:39:42.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.566+0000 7fb3f4f21640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-04T21:39:42.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.566+0000 7fb3f4f21640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-04-04T21:39:42.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.566+0000 7fb3f4f21640 1 -- 172.21.15.106:0/3624816212 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f0145780 con 0x7fb3f00a9a50 2024-04-04T21:39:42.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.568+0000 7fb3f4f21640 1 --2- 172.21.15.106:0/3624816212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3f00a9a50 0x7fb3f0146460 secure :-1 s=READY pgs=5560 cs=0 l=1 rev1=1 crypto rx=0x7fb3f01400d0 tx=0x7fb3e00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:42.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.568+0000 7fb3ed7fa640 10 client.?.objecter ms_handle_connect 0x7fb3f00a9a50 2024-04-04T21:39:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.568+0000 7fb3ed7fa640 10 client.?.objecter resend_mon_ops 2024-04-04T21:39:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3ed7fa640 1 -- 172.21.15.106:0/3624816212 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0032840 con 0x7fb3f00a9a50 2024-04-04T21:39:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3ed7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-04T21:39:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3ed7fa640 10 monclient: dump: 2024-04-04T21:39:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:42.575 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:42.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3ed7fa640 10 monclient: _finish_auth 0 2024-04-04T21:39:42.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3ed7fa640 10 monclient: _check_auth_tickets 2024-04-04T21:39:42.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-04T21:39:42.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3ed7fa640 1 -- 172.21.15.106:0/3624816212 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e0032a00 con 0x7fb3f00a9a50 2024-04-04T21:39:42.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-04-04T21:39:42.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3ed7fa640 1 -- 172.21.15.106:0/3624816212 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0032df0 con 0x7fb3f00a9a50 2024-04-04T21:39:42.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-04T21:39:42.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3ed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-04T21:39:42.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3ed7fa640 10 monclient: dump: 2024-04-04T21:39:42.575 INFO:teuthology.orchestra.run.smithi106.stderr:epoch 1 2024-04-04T21:39:42.575 INFO:teuthology.orchestra.run.smithi106.stderr:fsid 8f89ce43-1bcb-41cf-9555-c40eeb258155 2024-04-04T21:39:42.575 INFO:teuthology.orchestra.run.smithi106.stderr:last_changed 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr:created 2024-04-04T21:00:35.304825+0000 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr:min_mon_release 19 (squid) 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr:election_strategy: 1 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3f7185640 5 monclient: authenticate success, global_id 22153 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3f7185640 10 monclient: _renew_subs 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3f7185640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3eeffd640 4 set_mon_vals no callback set 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3f7185640 1 -- 172.21.15.106:0/3624816212 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f01459f0 con 0x7fb3f00a9a50 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3f7185640 10 client.22153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3f7185640 10 monclient: _renew_subs 2024-04-04T21:39:42.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3f7185640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:42.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3f7185640 1 -- 172.21.15.106:0/3624816212 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f013adf0 con 0x7fb3f00a9a50 2024-04-04T21:39:42.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.572+0000 7fb3f7185640 1 librados: init done 2024-04-04T21:39:42.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.573+0000 7fb3f7185640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:42.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.573+0000 7fb3f7185640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:42.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.573+0000 7fb3f7185640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:42.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.573+0000 7fb3f7185640 1 -- 172.21.15.106:0/3624816212 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f004f050 con 0x7fb3f00a9a50 2024-04-04T21:39:42.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.590+0000 7fb3ed7fa640 1 -- 172.21.15.106:0/3624816212 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb3e003a380 con 0x7fb3f00a9a50 2024-04-04T21:39:42.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.590+0000 7fb3ed7fa640 1 --2- 172.21.15.106:0/3624816212 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb3d006ceb0 0x7fb3d006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-04T21:39:42.591 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.591+0000 7fb3effff640 1 --2- 172.21.15.106:0/3624816212 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb3d006ceb0 0x7fb3d006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-04T21:39:42.591 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.591+0000 7fb3effff640 10 monclient: get_auth_request con 0x7fb3d006ceb0 auth_method 0 2024-04-04T21:39:42.591 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.591+0000 7fb3ed7fa640 10 client.22153.objecter ms_handle_connect 0x7fb3d006ceb0 2024-04-04T21:39:42.591 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.591+0000 7fb3effff640 1 --2- 172.21.15.106:0/3624816212 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb3d006ceb0 0x7fb3d006f370 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0136f30 tx=0x7fb3d8009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-04T21:39:42.592 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.592+0000 7fb3ed7fa640 1 -- 172.21.15.106:0/3624816212 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(1197..1197 src has 882..1197) v4 ==== 6353+0+0 (secure 0 0 0) 0x7fb3e002f070 con 0x7fb3f00a9a50 2024-04-04T21:39:42.592 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.592+0000 7fb3ed7fa640 10 client.22153.objecter ms_dispatch 0x7fb3f0000b90 osd_map(1197..1197 src has 882..1197) v4 2024-04-04T21:39:42.592 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.592+0000 7fb3ed7fa640 3 client.22153.objecter handle_osd_map got epochs [1197,1197] > 0 2024-04-04T21:39:42.592 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.592+0000 7fb3ed7fa640 3 client.22153.objecter handle_osd_map decoding full epoch 1197 2024-04-04T21:39:42.592 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.592+0000 7fb3ed7fa640 20 client.22153.objecter dump_active .. 0 homeless 2024-04-04T21:39:42.828 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.828+0000 7fb3ed7fa640 1 -- 172.21.15.106:0/3624816212 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb3f01459f0 con 0x7fb3f00a9a50 2024-04-04T21:39:42.828 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.828+0000 7fb3ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-04T21:39:42.828 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:42.828+0000 7fb3ed7fa640 10 monclient: _finish_command 1 = system:0 2024-04-04T21:39:43.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.008+0000 7fb3f7185640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-04T21:39:43.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.008+0000 7fb3f7185640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-04T21:39:43.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.008+0000 7fb3f7185640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-04-04T21:39:43.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.008+0000 7fb3f7185640 1 -- 172.21.15.106:0/3624816212 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fb3f00b7c60 con 0x7fb3f00a9a50 2024-04-04T21:39:43.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.195+0000 7fb3ed7fa640 1 -- 172.21.15.106:0/3624816212 <== mon.1 v2:172.21.15.203:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fb3e0037020 con 0x7fb3f00a9a50 2024-04-04T21:39:43.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.195+0000 7fb3ed7fa640 10 monclient: handle_config config(1 keys) v1 2024-04-04T21:39:43.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.195+0000 7fb3eeffd640 4 set_mon_vals no callback set 2024-04-04T21:39:43.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.195+0000 7fb3eeffd640 10 set_mon_vals mon_health_to_clog = false 2024-04-04T21:39:43.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.195+0000 7fb3ed7fa640 1 -- 172.21.15.106:0/3624816212 <== mon.1 v2:172.21.15.203:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) v1 ==== 123+0+0 (secure 0 0 0) 0x7fb3e0040070 con 0x7fb3f00a9a50 2024-04-04T21:39:43.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.195+0000 7fb3ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-04T21:39:43.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.195+0000 7fb3ed7fa640 10 monclient: _finish_command 2 = system:0 2024-04-04T21:39:43.198 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 10 librados: watch_flush enter 2024-04-04T21:39:43.198 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 10 librados: watch_flush exit 2024-04-04T21:39:43.198 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 20 client.22153.objecter shutdown clearing up homeless session... 2024-04-04T21:39:43.198 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 10 client.22153.objecter successfully canceled tick 2024-04-04T21:39:43.198 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 1 -- 172.21.15.106:0/3624816212 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb3d006ceb0 msgr2=0x7fb3d006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:43.198 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 1 --2- 172.21.15.106:0/3624816212 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb3d006ceb0 0x7fb3d006f370 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0136f30 tx=0x7fb3d8009040 comp rx=0 tx=0).stop 2024-04-04T21:39:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 10 monclient: shutdown 2024-04-04T21:39:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-04T21:39:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 1 -- 172.21.15.106:0/3624816212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3f00a9a50 msgr2=0x7fb3f0146460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-04T21:39:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 1 --2- 172.21.15.106:0/3624816212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3f00a9a50 0x7fb3f0146460 secure :-1 s=READY pgs=5560 cs=0 l=1 rev1=1 crypto rx=0x7fb3f01400d0 tx=0x7fb3e00046c0 comp rx=0 tx=0).stop 2024-04-04T21:39:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 1 -- 172.21.15.106:0/3624816212 shutdown_connections 2024-04-04T21:39:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 1 --2- 172.21.15.106:0/3624816212 >> [v2:172.21.15.106:6800/2457463546,v1:172.21.15.106:6801/2457463546] conn(0x7fb3d006ceb0 0x7fb3d006f370 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 1 --2- 172.21.15.106:0/3624816212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3f013fac0 0x7fb3f013a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 1 --2- 172.21.15.106:0/3624816212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3f013f6a0 0x7fb3f01dbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 1 --2- 172.21.15.106:0/3624816212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3f00a9a50 0x7fb3f0146460 unknown :-1 s=CLOSED pgs=5560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-04T21:39:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 1 -- 172.21.15.106:0/3624816212 >> 172.21.15.106:0/3624816212 conn(0x7fb3f00b8030 msgr2=0x7fb3f00b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-04T21:39:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 1 -- 172.21.15.106:0/3624816212 shutdown_connections 2024-04-04T21:39:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 1 -- 172.21.15.106:0/3624816212 wait complete. 2024-04-04T21:39:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-04-04T21:39:43.197+0000 7fb3f7185640 1 librados: shutdown 2024-04-04T21:39:43.209 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-04T21:39:43.209 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-04T21:39:43.209 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-04T21:39:43.209 INFO:teuthology.orchestra.run:waiting for 300 2024-04-04T21:39:43.222 INFO:tasks.ceph.osd.0.smithi106.stderr:2024-04-04T21:39:43.213+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35474) UID: 0 2024-04-04T21:39:43.222 INFO:tasks.ceph.osd.0.smithi106.stderr:2024-04-04T21:39:43.214+0000 b47a640 -1 osd.0 1197 *** Got signal Terminated *** 2024-04-04T21:39:46.043 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:37.560 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:46.046 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:37.563 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:46.062 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:37.579 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:46.090 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:37.572 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:46.092 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:37.608 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:46.092 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:37.575 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:46.093 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:37.610 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:46.103 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:37.586 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:46.105 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:37.621 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:51.300 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:42.817 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:51.350 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:42.866 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:51.371 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:42.888 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:51.402 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:42.885 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:51.404 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:42.887 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:51.406 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:42.922 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:51.407 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:42.924 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:51.418 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:42.901 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:39:51.420 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:42.937 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:01.074 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:52.591 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:01.094 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:52.612 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:01.105 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:52.622 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:01.125 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:52.642 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:01.126 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:52.643 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:01.127 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:52.610 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:01.128 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:52.611 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:01.138 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:52.655 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:01.139 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:52.622 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:06.093 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:57.610 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:06.095 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:57.612 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:06.104 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:38:57.622 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:06.140 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:57.657 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:06.141 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:57.624 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:06.142 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:57.625 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:06.144 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:57.661 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:06.152 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:38:57.635 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:06.155 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:38:57.672 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:21.094 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:12.611 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:21.144 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:12.661 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:21.156 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:12.674 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:21.194 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:12.676 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:21.197 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:12.680 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:21.197 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:12.714 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:21.201 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:12.718 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:21.211 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:12.693 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:21.217 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:12.734 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:26.138 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:17.655 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:26.142 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:17.659 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:26.146 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:17.629 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:26.147 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:17.630 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:26.153 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:17.670 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:26.157 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:17.640 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:26.187 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:17.704 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:26.188 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:17.705 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:26.198 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:17.715 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:30.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-04T21:40:32.022 INFO:tasks.ceph.osd.0:Stopped 2024-04-04T21:40:32.023 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-04T21:40:32.023 INFO:teuthology.orchestra.run:waiting for 300 2024-04-04T21:40:32.036 INFO:tasks.ceph.osd.1.smithi106.stderr:2024-04-04T21:40:32.027+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35476) UID: 0 2024-04-04T21:40:32.036 INFO:tasks.ceph.osd.1.smithi106.stderr:2024-04-04T21:40:32.028+0000 b47a640 -1 osd.1 1203 *** Got signal Terminated *** 2024-04-04T21:40:36.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-04T21:40:41.189 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:32.706 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:41.191 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:32.709 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:41.211 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:32.729 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:41.239 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:32.756 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:41.243 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:32.760 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:41.245 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:32.728 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:41.246 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:32.729 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:41.265 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:32.781 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:41.266 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:32.749 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:41.954 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:33.436 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:41.954 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:33.437 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:41.963 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:33.446 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:41.980 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:33.498 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:41.981 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:33.499 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:41.992 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:33.510 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:42.009 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:33.526 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:42.010 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:33.527 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:42.022 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:33.539 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:40:42.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-04T21:40:44.225 INFO:tasks.ceph.osd.1:Stopped 2024-04-04T21:40:44.226 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-04T21:40:44.226 INFO:teuthology.orchestra.run:waiting for 300 2024-04-04T21:40:44.239 INFO:tasks.ceph.osd.2.smithi106.stderr:2024-04-04T21:40:44.230+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35483) UID: 0 2024-04-04T21:40:44.242 INFO:tasks.ceph.osd.2.smithi106.stderr:2024-04-04T21:40:44.231+0000 b47a640 -1 osd.2 1205 *** Got signal Terminated *** 2024-04-04T21:40:48.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-04T21:41:00.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-04T21:41:01.264 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:52.781 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:01.265 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:52.782 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:01.266 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:52.749 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:01.271 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:52.754 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:01.286 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:52.803 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:01.291 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:52.774 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:01.313 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:52.830 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:01.315 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:52.832 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:01.334 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:52.851 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:01.979 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:53.461 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:01.980 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:53.462 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:01.988 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:39:53.470 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:02.029 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:53.546 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:02.030 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:53.547 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:02.044 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:39:53.561 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:02.048 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:53.565 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:02.049 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:53.566 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:02.058 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:39:53.574 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:02.530 INFO:tasks.ceph.osd.2:Stopped 2024-04-04T21:41:02.531 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-04T21:41:02.531 INFO:teuthology.orchestra.run:waiting for 300 2024-04-04T21:41:02.549 INFO:tasks.ceph.osd.3.smithi106.stderr:2024-04-04T21:41:02.536+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35491) UID: 0 2024-04-04T21:41:02.552 INFO:tasks.ceph.osd.3.smithi106.stderr:2024-04-04T21:41:02.538+0000 b47a640 -1 osd.3 1209 *** Got signal Terminated *** 2024-04-04T21:41:06.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-04T21:41:16.380 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:07.897 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:16.398 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:07.916 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:16.461 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:07.977 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:16.462 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:07.979 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:16.473 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:07.956 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:16.477 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:07.960 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:16.491 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:08.008 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:16.494 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:07.976 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:16.513 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:08.029 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:21.353 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:12.870 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:21.354 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:12.871 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:21.370 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:12.888 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:21.404 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:12.886 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:21.404 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:12.920 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:21.405 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:12.888 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:21.405 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:12.922 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:21.419 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:12.936 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:21.420 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:12.902 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:26.936 INFO:tasks.ceph.osd.3:Stopped 2024-04-04T21:41:26.937 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-04T21:41:26.937 INFO:teuthology.orchestra.run:waiting for 300 2024-04-04T21:41:26.950 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:41:26.942+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35353) UID: 0 2024-04-04T21:41:26.951 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-04T21:41:26.943+0000 b47a640 -1 osd.4 1213 *** Got signal Terminated *** 2024-04-04T21:41:30.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-04-04T21:41:31.386 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:22.903 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:31.389 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:22.906 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:31.407 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:22.924 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:31.437 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:22.919 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:31.437 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:22.954 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:31.438 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:22.921 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:31.439 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:22.955 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:31.451 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:22.968 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:31.451 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:22.934 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:33.041 INFO:tasks.ceph.osd.4:Stopped 2024-04-04T21:41:33.041 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-04T21:41:33.042 INFO:teuthology.orchestra.run:waiting for 300 2024-04-04T21:41:33.057 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-04T21:41:33.046+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35356) UID: 0 2024-04-04T21:41:33.057 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-04T21:41:33.047+0000 b47a640 -1 osd.5 1215 *** Got signal Terminated *** 2024-04-04T21:41:36.376 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-04-04T21:41:41.413 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:32.930 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:41.418 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:32.935 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:41.432 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:32.949 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:41.461 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:32.944 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:41.462 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:32.945 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:41.474 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:32.957 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:41.482 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:32.998 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:41.483 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:33.000 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:41.501 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:33.017 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:51.438 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:42.955 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:51.441 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:42.958 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:51.451 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:42.969 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:51.486 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:42.968 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:51.486 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:43.003 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:51.487 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:42.970 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:51.487 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:43.004 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:51.498 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:42.981 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:41:51.499 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:43.016 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:01.455 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:52.972 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:01.500 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:53.017 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:01.511 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:40:53.028 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:01.547 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:53.030 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:01.548 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:53.064 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:01.549 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:53.031 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:01.549 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:53.065 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:01.560 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:40:53.077 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:01.564 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:40:53.047 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:09.651 INFO:tasks.ceph.osd.5:Stopped 2024-04-04T21:42:09.651 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-04T21:42:09.651 INFO:teuthology.orchestra.run:waiting for 300 2024-04-04T21:42:09.665 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-04-04T21:42:09.656+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35355) UID: 0 2024-04-04T21:42:09.667 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-04-04T21:42:09.657+0000 b47a640 -1 osd.6 1217 *** Got signal Terminated *** 2024-04-04T21:42:11.502 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:03.019 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:11.504 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:03.021 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:11.520 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:03.038 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:11.550 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:03.033 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:11.551 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:03.034 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:11.556 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:03.073 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:11.557 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:03.074 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:11.562 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:03.045 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:11.569 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:03.086 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:21.526 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:13.043 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:21.530 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:13.047 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:21.542 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:13.060 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:21.575 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:13.057 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:21.576 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:13.059 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:21.587 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:13.070 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:21.598 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:13.114 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:21.599 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:13.116 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:21.610 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:13.127 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:31.557 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:23.074 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:31.559 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:23.076 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:31.571 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:23.088 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:31.603 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:23.120 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:31.605 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:23.121 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:31.605 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:23.087 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:31.606 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:23.088 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:31.617 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:23.100 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:31.619 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:23.136 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:41.589 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:33.106 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:41.590 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:33.108 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:41.603 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:33.120 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:41.635 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:33.152 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:41.636 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:33.153 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:41.638 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:33.121 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:41.639 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:33.122 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:41.648 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:33.165 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:41.650 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:33.133 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:51.615 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:43.132 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:51.617 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:43.134 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:51.630 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:43.147 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:51.660 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:43.177 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:51.661 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:43.144 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:51.662 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:43.145 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:51.663 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:43.180 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:51.674 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:43.157 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:42:51.675 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:43.192 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:01.638 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:53.154 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:01.640 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:53.157 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:01.653 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:41:53.170 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:01.683 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:53.200 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:01.687 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:53.203 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:01.687 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:53.169 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:01.688 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:53.171 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:01.698 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:41:53.215 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:01.699 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:41:53.182 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:11.620 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:03.137 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:11.667 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:03.150 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:11.668 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:03.185 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:11.670 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:03.153 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:11.681 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:03.164 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:11.682 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:03.199 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:11.715 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:03.232 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:11.716 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:03.233 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:11.728 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:03.245 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:17.546 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:09.063 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:17.551 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:09.069 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:17.566 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:09.083 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:17.604 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:09.120 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:17.604 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:09.086 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:17.605 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:09.088 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:17.606 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:09.122 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:17.619 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:09.101 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:17.621 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:09.138 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:21.645 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:13.162 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:21.691 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:13.209 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:21.692 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:13.174 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:21.695 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:13.178 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:21.705 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:13.188 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:21.707 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:13.225 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:21.742 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:13.258 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:21.745 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:13.262 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:21.756 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:13.272 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:31.717 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:23.232 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:31.719 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:23.236 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:31.730 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:23.247 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:31.765 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:23.247 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:31.766 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:23.282 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:31.767 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:23.284 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:31.768 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:23.250 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:31.779 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:23.261 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:31.780 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:23.297 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:33.753 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:33.749+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:34.776 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:34.774+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:35.734 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:35.733+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:36.710 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:36.701+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:37.713 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:37.713+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:38.752 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:38.751+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:39.738 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:39.737+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:40.749 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:40.747+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:41.702 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:41.700+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:41.741 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:33.258 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:41.745 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:33.262 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:41.758 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:33.275 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:41.790 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:33.272 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:41.791 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:33.273 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:41.794 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:33.310 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:41.795 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:33.312 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:41.802 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:33.285 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:41.807 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:33.324 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:42.753 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:42.751+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:43.725 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:43.723+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:44.699 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:44.697+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:45.695 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:45.693+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:46.658 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:46.657+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:47.628 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:47.626+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:48.590 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:48.587+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:49.603 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:49.602+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:50.601 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:50.599+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:51.556 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:51.554+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:51.725 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:43.242 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:51.774 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:43.291 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:51.775 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:43.292 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:51.778 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:43.295 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:51.792 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:43.309 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:51.793 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:43.310 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:51.819 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:43.301 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:51.820 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:43.302 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:51.831 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:43.314 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:43:52.562 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:52.561+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:53.576 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:53.574+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:54.623 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:54.622+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:55.590 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:55.589+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:56.592 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:56.592+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:57.574 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:57.573+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:58.577 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:58.576+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:43:59.570 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:43:59.569+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:00.620 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:00.619+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:01.611 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:01.610+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:01.752 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:53.269 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:01.800 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:53.317 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:01.801 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:53.283 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:01.803 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:53.285 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:01.814 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:42:53.331 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:01.814 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:42:53.297 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:01.846 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:53.363 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:01.848 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:53.365 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:01.859 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:42:53.376 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:02.651 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:02.650+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:03.617 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:03.617+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:04.620 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:04.619+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:05.656 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:05.655+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:06.629 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:06.629+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:07.676 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:07.675+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:08.681 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:08.680+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:09.639 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:09.639+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:10.644 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:10.643+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:11.616 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:11.614+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:11.826 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:03.343 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:11.829 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:03.346 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:11.840 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:03.357 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:11.876 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:03.358 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:11.876 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:03.393 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:11.877 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:03.360 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:11.877 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:03.394 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:11.888 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:03.371 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:11.889 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:03.406 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:12.651 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:12.651+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:13.658 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:13.656+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:14.702 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:14.697+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:15.724 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:15.724+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:16.718 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:16.717+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:17.740 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:17.739+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:18.706 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:18.704+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:19.657 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:19.656+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:20.696 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:20.694+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:21.684 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:21.681+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:21.846 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:13.363 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:21.850 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:13.368 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:21.863 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:13.380 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:21.895 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:13.378 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:21.896 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:13.412 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:21.896 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:13.379 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:21.897 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:13.414 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:21.907 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:13.390 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:21.908 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:13.425 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:22.718 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:22.716+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:23.759 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:23.757+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:24.761 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:24.758+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:25.738 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:25.735+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:26.781 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:26.780+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:27.762 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:27.760+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:28.764 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:28.762+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:29.743 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:29.741+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:30.737 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:30.736+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:31.714 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:31.713+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:31.876 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:23.393 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:31.878 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:23.396 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:31.891 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:23.408 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:31.926 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:23.408 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:31.927 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:23.410 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:31.927 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:23.444 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:31.928 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:23.445 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:31.938 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:23.421 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:31.944 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:23.461 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:32.750 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:32.748+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:33.757 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:33.756+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:34.790 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:34.789+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:35.830 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:35.829+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:36.813 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:36.811+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:37.843 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:37.842+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:38.886 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:38.886+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:39.911 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:39.910+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:40.946 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:40.946+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:41.904 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:33.421 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:41.908 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:33.426 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:41.921 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:33.438 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:41.955 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:33.472 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:41.956 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:33.438 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:41.957 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:33.440 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:41.958 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:33.475 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:41.970 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:33.452 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:41.970 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:33.487 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:41.983 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:41.981+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:42.970 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:42.969+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:43.950 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:43.950+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:44.954 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:44.953+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:45.966 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:45.965+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:46.960 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:46.959+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:47.920 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:47.919+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:48.877 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:48.876+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:49.873 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:49.872+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:50.894 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:50.893+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:51.878 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:51.876+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:51.934 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:43.452 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:51.936 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:43.454 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:51.948 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:43.465 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:51.983 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:43.499 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:51.983 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:43.466 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:51.984 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:43.501 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:51.985 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:43.468 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:51.995 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:43.512 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:51.996 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:43.479 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:44:52.920 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:52.920+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:53.915 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:53.915+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:54.959 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:54.958+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:55.944 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:55.943+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:56.988 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:56.987+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:58.030 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:58.029+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:58.982 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:58.980+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:44:59.989 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:44:59.988+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:45:00.987 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:45:00.986+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:45:01.954 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:53.470 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:01.956 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:53.474 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:01.970 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:43:53.488 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:02.003 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:53.486 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:02.004 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:53.487 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:02.005 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:45:02.003+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:45:02.012 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:53.529 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:02.014 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:53.530 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:02.016 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:43:53.498 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:02.025 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:43:53.542 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:02.963 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:45:02.961+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:45:04.009 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:45:04.007+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:45:05.013 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:45:05.011+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:45:06.050 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:45:06.049+0000 ea52640 -1 osd.7 1217 heartbeat_check: no reply from 172.21.15.203:6820 osd.6 since back 2024-04-04T21:42:08.919116+0000 front 2024-04-04T21:42:08.920756+0000 (oldest deadline 2024-04-04T21:43:33.617826+0000) 2024-04-04T21:45:11.983 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:03.500 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:11.986 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:03.503 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:11.997 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:03.515 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:12.034 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:03.516 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:12.035 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:03.548 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:12.035 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:03.549 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:12.035 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:03.518 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:12.044 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:03.561 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:12.046 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:03.529 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:22.012 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:13.529 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:22.015 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:13.533 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:22.028 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:13.545 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:22.060 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:13.543 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:22.063 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:13.546 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:22.063 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:13.580 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:22.066 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:13.583 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:22.075 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:13.557 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:22.078 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:13.595 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:32.067 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:23.585 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:32.072 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:23.590 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:32.092 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:23.610 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:32.117 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:23.600 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:32.120 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:23.603 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:32.126 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:23.643 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:32.128 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:23.645 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:32.133 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:23.615 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:32.143 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:23.660 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:42.099 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:33.616 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:42.102 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:33.620 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:42.114 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:33.631 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:42.146 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:33.629 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:42.148 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:33.630 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:42.149 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:33.666 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:42.151 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:33.668 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:42.159 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:33.641 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:42.162 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:33.679 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:47.839 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-04-04T21:45:49.303 INFO:tasks.ceph.osd.6:Stopped 2024-04-04T21:45:49.303 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-04T21:45:49.303 INFO:teuthology.orchestra.run:waiting for 300 2024-04-04T21:45:49.317 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:45:49.307+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35359) UID: 0 2024-04-04T21:45:49.317 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-04-04T21:45:49.309+0000 b47a640 -1 osd.7 1217 *** Got signal Terminated *** 2024-04-04T21:45:52.129 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:43.646 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:52.131 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:43.648 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:52.142 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:43.659 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:52.174 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:43.656 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:52.175 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:43.657 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:52.177 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:43.694 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:52.178 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:43.695 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:52.187 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:43.669 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:52.191 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:43.707 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:45:53.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-04-04T21:46:02.152 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:53.668 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:02.156 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:53.674 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:02.169 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:44:53.686 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:02.203 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:53.686 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:02.204 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:53.687 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:02.216 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:44:53.699 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:02.224 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:53.741 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:02.225 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:53.742 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:02.237 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:44:53.754 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:12.184 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:03.700 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:12.186 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:03.703 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:12.198 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:03.715 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:12.233 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:03.716 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:12.234 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:03.717 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:12.235 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:03.751 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:12.236 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:03.752 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:12.245 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:03.728 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:12.247 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:03.764 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:22.259 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:13.776 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:22.265 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:13.783 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:22.309 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:13.792 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:22.312 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:13.795 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:22.313 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:13.830 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:22.313 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:13.830 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:22.328 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:13.845 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:22.330 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:13.812 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:22.357 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:13.873 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:27.259 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:18.776 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:27.265 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:18.782 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:27.280 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:18.798 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:27.310 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:18.793 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:27.312 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:18.829 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:27.314 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:18.831 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:27.315 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:18.798 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:27.326 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:18.809 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:27.327 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:18.843 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:42.294 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:33.811 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:42.299 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:33.816 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:42.314 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:33.832 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:42.346 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:33.828 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:42.346 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:33.863 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:42.347 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:33.829 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:42.347 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:33.864 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:42.362 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:33.844 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:42.362 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:33.879 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:47.313 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:38.830 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:47.317 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:38.835 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:47.331 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:38.848 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:47.361 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:38.844 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:47.364 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:38.881 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:47.366 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:38.848 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:47.367 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:38.884 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:47.377 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:38.859 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:46:47.379 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:38.896 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:00.560 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:52.077 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:00.579 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:52.096 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:00.598 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:52.116 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:00.612 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:52.094 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:00.613 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:52.096 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:00.616 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:52.133 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:00.618 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:52.134 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:00.629 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:52.112 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:00.634 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:52.151 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:02.345 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:53.862 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:02.350 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:53.868 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:02.360 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:53.877 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:02.394 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:53.877 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:02.396 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:53.878 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:02.397 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:53.914 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:02.398 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:53.915 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:02.405 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:53.887 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:02.410 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:53.927 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:07.363 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:58.880 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:07.369 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:58.886 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:07.383 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:45:58.900 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:07.417 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:58.934 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:07.419 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:58.936 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:07.426 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:58.909 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:07.427 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:58.910 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:07.430 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:45:58.947 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:07.437 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:45:58.920 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:22.453 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:13.969 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:22.455 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:13.972 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:22.470 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:13.987 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:22.503 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:13.985 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:22.504 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:13.987 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:22.518 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:14.035 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:22.519 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:14.002 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:22.519 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:14.036 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:22.534 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:14.051 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:27.465 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:18.982 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:27.472 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:18.989 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:27.482 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:18.999 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:27.513 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:18.995 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:27.514 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:18.997 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:27.515 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:19.032 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:27.517 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:19.034 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:27.525 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:19.007 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:27.531 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:19.048 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:42.581 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:34.098 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:42.610 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:34.128 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:42.625 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:34.142 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:42.632 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:34.115 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:42.633 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:34.116 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:42.642 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:34.159 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:42.644 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:34.161 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:42.649 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:34.131 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:42.665 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:34.182 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:47.522 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:39.039 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:47.523 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:39.005 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:47.524 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:39.041 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:47.524 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:39.007 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:47.526 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:39.043 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:47.528 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:39.045 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:47.536 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:39.019 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:47.538 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:39.055 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:47:47.540 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:39.057 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:02.588 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:54.104 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:02.589 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:54.106 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:02.603 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:54.120 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:02.637 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:54.120 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:02.639 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:54.156 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:02.640 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:54.157 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:02.641 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:54.124 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:02.654 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:54.171 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:02.655 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:54.137 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:07.597 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:59.114 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:07.598 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:59.116 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:07.608 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:46:59.126 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:07.644 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:59.161 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:07.644 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:59.127 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:07.645 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:59.128 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:07.647 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:59.163 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:07.655 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:46:59.138 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:07.657 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:46:59.174 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:22.651 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:14.168 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:22.654 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:14.171 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:22.667 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:14.184 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:22.700 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:14.183 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:22.701 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:14.184 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:22.703 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:14.220 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:22.704 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:14.221 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:22.715 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:14.197 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:22.718 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:14.235 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:27.662 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:19.179 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:27.666 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:19.183 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:27.676 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:19.193 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:27.708 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:19.191 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:27.710 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:19.192 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:27.712 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:19.229 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:27.713 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:19.230 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:27.719 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:19.202 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:27.727 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:19.244 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:42.701 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:34.218 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:42.705 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:34.223 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:42.722 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:34.236 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:42.754 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:34.270 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:42.754 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:34.237 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:42.755 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:34.272 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:42.755 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:34.238 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:42.771 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:34.253 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:42.772 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:34.289 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:47.716 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:39.233 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:47.718 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:39.235 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:47.728 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:39.245 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:47.764 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:39.281 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:47.764 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:39.247 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:47.765 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:39.282 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:47.766 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:39.248 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:47.776 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:39.258 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:48:47.777 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:39.294 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:02.754 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:54.271 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:02.757 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:54.275 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:02.771 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:54.288 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:02.803 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:54.285 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:02.804 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:54.287 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:02.818 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:54.300 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:02.819 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:54.336 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:02.821 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:54.338 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:02.835 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:54.352 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:07.766 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:59.283 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:07.771 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:59.289 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:07.781 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:47:59.298 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:07.816 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:59.298 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:07.816 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:59.333 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:07.817 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:59.299 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:07.818 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:59.335 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:07.826 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:47:59.309 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:07.828 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:47:59.345 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:22.808 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:14.325 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:22.810 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:14.327 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:22.824 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:14.342 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:22.858 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:14.340 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:22.859 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:14.341 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:22.860 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:14.376 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:22.861 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:14.378 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:22.873 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:14.356 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:22.882 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:14.399 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:27.824 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:19.341 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:27.825 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:19.343 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:27.835 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:19.353 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:27.870 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:19.353 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:27.871 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:19.387 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:27.871 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:19.354 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:27.872 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:19.389 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:27.881 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:19.364 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:27.886 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:19.403 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:42.877 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:34.394 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:42.881 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:34.399 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:42.896 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:34.413 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:42.930 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:34.412 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:42.930 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:34.447 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:42.931 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:34.414 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:42.931 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:34.448 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:42.945 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:34.428 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:42.946 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:34.462 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:48.033 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:39.550 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:48.035 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:39.552 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:48.045 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:39.563 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:48.079 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:39.596 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:48.081 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:39.597 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:48.081 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:39.564 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:48.084 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:39.567 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:48.092 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:39.608 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:49:48.095 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:39.577 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:02.954 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:54.471 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:02.958 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:54.475 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:02.991 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:54.508 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:03.004 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:54.486 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:03.005 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:54.487 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:03.012 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:54.529 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:03.013 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:54.530 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:03.020 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:54.503 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:03.037 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:54.553 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:07.964 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:59.481 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:07.968 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:59.485 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:07.978 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:48:59.496 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:08.014 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:59.531 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:08.014 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:59.497 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:08.016 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:59.498 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:08.016 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:59.533 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:08.026 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:48:59.508 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:08.027 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:48:59.543 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:22.957 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:14.475 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:23.008 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:14.490 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:23.008 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:14.526 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:23.011 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:14.494 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:23.024 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:14.541 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:23.025 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:14.507 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:23.054 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:14.571 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:23.055 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:14.572 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:23.070 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:14.587 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:27.974 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:19.491 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:28.023 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:19.505 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:28.025 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:19.542 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:28.025 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:19.507 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:28.035 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:19.517 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:28.036 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:19.554 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:28.075 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:19.591 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:28.075 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:19.592 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:28.086 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:19.603 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.152 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:34.670 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.220 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:34.737 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.246 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:34.763 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.261 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:34.743 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.264 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:34.747 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.278 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:34.795 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.281 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:34.763 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.281 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:34.797 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.299 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:34.816 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.955 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:35.437 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.956 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:35.439 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.960 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:35.478 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.961 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:35.479 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.968 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:35.450 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.974 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:35.491 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.975 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:35.492 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.975 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:35.493 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:43.987 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:35.504 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:48.220 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:39.737 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:48.221 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:39.738 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:48.232 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:39.750 35186== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:48.266 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:39.748 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:48.267 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:39.749 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:48.269 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:39.786 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:48.270 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:39.787 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:48.278 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:39.760 34818== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:48.281 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:39.798 35188== Warning: unimplemented fcntl command: 1036 2024-04-04T21:50:48.376 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-04T21:50:48.376 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-04T21:50:48.376 INFO:teuthology.orchestra.run:waiting for 300 2024-04-04T21:50:48.377 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:50:48.377+0000 7f6db6429640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 35193) UID: 0 2024-04-04T21:50:48.377 INFO:tasks.ceph.mgr.y.smithi106.stderr:2024-04-04T21:50:48.377+0000 7f6db6429640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-04T21:50:48.403 INFO:tasks.ceph.mgr.y:Stopped 2024-04-04T21:50:48.403 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-04T21:50:48.404 INFO:teuthology.orchestra.run:waiting for 300 2024-04-04T21:50:48.428 INFO:tasks.ceph.mgr.x:Stopped 2024-04-04T21:50:48.428 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-04T21:50:48.428 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-04T21:50:48.428 INFO:teuthology.orchestra.run:waiting for 300 2024-04-04T21:50:48.464 INFO:tasks.ceph.mon.a.smithi106.stderr:2024-04-04T21:50:48.453+0000 9f8e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35153) UID: 0 2024-04-04T21:50:48.465 INFO:tasks.ceph.mon.a.smithi106.stderr:2024-04-04T21:50:48.454+0000 9f8e640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-04T21:50:49.460 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:40.977 35186== 2024-04-04T21:50:49.460 INFO:tasks.ceph.mon.a.smithi106.stderr:==00:00:49:40.977 35186== Exit program on first error (--exit-on-first-error=yes) 2024-04-04T21:50:49.485 INFO:tasks.ceph.mon.a.smithi106.stderr:daemon-helper: command failed with exit status 42 2024-04-04T21:50:50.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-04-04T21:50:54.529 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-04-04T21:50:54.529 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi106 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-04-04T21:50:54.532 INFO:tasks.ceph.mon.a:Stopped 2024-04-04T21:50:54.532 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-04T21:50:54.532 INFO:teuthology.orchestra.run:waiting for 300 2024-04-04T21:50:54.547 INFO:tasks.ceph.mon.c.smithi106.stderr:2024-04-04T21:50:54.537+0000 9f8e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35155) UID: 0 2024-04-04T21:50:54.548 INFO:tasks.ceph.mon.c.smithi106.stderr:2024-04-04T21:50:54.538+0000 9f8e640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-04-04T21:50:55.276 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:46.792 35188== 2024-04-04T21:50:55.276 INFO:tasks.ceph.mon.c.smithi106.stderr:==00:00:49:46.792 35188== Exit program on first error (--exit-on-first-error=yes) 2024-04-04T21:50:55.296 INFO:tasks.ceph.mon.c.smithi106.stderr:daemon-helper: command failed with exit status 42 2024-04-04T21:50:55.415 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-04-04T21:50:55.415 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-04-04T21:51:00.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~5s 2024-04-04T21:51:00.633 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-04-04T21:51:00.633 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi106 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-04-04T21:51:00.634 INFO:tasks.ceph.mon.c:Stopped 2024-04-04T21:51:00.634 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-04T21:51:00.635 INFO:teuthology.orchestra.run:waiting for 300 2024-04-04T21:51:00.650 INFO:tasks.ceph.mon.b.smithi203.stderr:2024-04-04T21:51:00.638+0000 9f8e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34814) UID: 0 2024-04-04T21:51:00.653 INFO:tasks.ceph.mon.b.smithi203.stderr:2024-04-04T21:51:00.640+0000 9f8e640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-04-04T21:51:01.446 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:52.926 34818== 2024-04-04T21:51:01.446 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:49:52.926 34818== Exit program on first error (--exit-on-first-error=yes) 2024-04-04T21:51:01.464 INFO:tasks.ceph.mon.b.smithi203.stderr:daemon-helper: command failed with exit status 42 2024-04-04T21:51:05.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-04-04T21:51:05.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-04-04T21:51:06.736 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-04-04T21:51:06.737 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-04-04T21:51:06.737 INFO:tasks.ceph.mon.b:Stopped 2024-04-04T21:51:06.737 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-04T21:51:06.738 DEBUG:teuthology.orchestra.run.smithi106:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-04-04T21:51:06.834 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi106.front.sepia.ceph.com 2024-04-04T21:51:06.834 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-04T21:51:06.899 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi106.front.sepia.ceph.com 2024-04-04T21:51:06.899 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-04T21:51:06.958 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi106.front.sepia.ceph.com 2024-04-04T21:51:06.958 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-04T21:51:07.016 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi106.front.sepia.ceph.com 2024-04-04T21:51:07.016 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-04T21:51:07.073 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-04T21:51:07.073 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-04T21:51:07.217 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-04T21:51:07.217 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-04T21:51:07.281 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-04T21:51:07.281 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-04T21:51:07.338 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-04T21:51:07.338 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-04T21:51:07.397 INFO:tasks.ceph:Archiving mon data... 2024-04-04T21:51:07.398 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121/data/mon.a.tgz 2024-04-04T21:51:07.398 DEBUG:teuthology.orchestra.run.smithi106:> mktemp 2024-04-04T21:51:07.414 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.bl9oAjJPTk 2024-04-04T21:51:07.415 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.bl9oAjJPTk 2024-04-04T21:51:07.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.bl9oAjJPTk 2024-04-04T21:51:07.661 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.bl9oAjJPTk is 471KB 2024-04-04T21:51:07.761 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.bl9oAjJPTk 2024-04-04T21:51:07.775 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121/data/mon.c.tgz 2024-04-04T21:51:07.775 DEBUG:teuthology.orchestra.run.smithi106:> mktemp 2024-04-04T21:51:07.829 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.ziIgkXptHc 2024-04-04T21:51:07.829 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ziIgkXptHc 2024-04-04T21:51:07.979 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.ziIgkXptHc 2024-04-04T21:51:08.064 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.ziIgkXptHc is 471KB 2024-04-04T21:51:08.163 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.ziIgkXptHc 2024-04-04T21:51:08.178 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121/data/mon.b.tgz 2024-04-04T21:51:08.178 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-04-04T21:51:08.193 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.bIxEWDmSfq 2024-04-04T21:51:08.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.bIxEWDmSfq 2024-04-04T21:51:08.367 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.bIxEWDmSfq 2024-04-04T21:51:08.451 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.bIxEWDmSfq is 471KB 2024-04-04T21:51:08.546 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.bIxEWDmSfq 2024-04-04T21:51:08.560 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-04T21:51:08.560 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-04T21:51:08.563 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-04T21:51:08.915 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-04-04T21:51:09.172 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-04-04T21:51:09.174 INFO:tasks.ceph:Archiving crash dumps... 2024-04-04T21:51:09.176 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121/remote/smithi106/crash 2024-04-04T21:51:09.177 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-04T21:51:09.203 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121/remote/smithi203/crash 2024-04-04T21:51:09.204 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-04T21:51:09.230 INFO:tasks.ceph:Compressing logs... 2024-04-04T21:51:09.230 DEBUG:teuthology.orchestra.run.smithi106:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-04T21:51:09.248 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-04T21:51:09.278 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108947.log 2024-04-04T21:51:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42111.log 2024-04-04T21:51:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.108947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41351.log 2024-04-04T21:51:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.42111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35603.log 2024-04-04T21:51:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41351.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108947.log.gz 2024-04-04T21:51:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40392.log 2024-04-04T21:51:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.35603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42368.log 2024-04-04T21:51:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40816.log 2024-04-04T21:51:09.279 INFO:teuthology.orchestra.run.smithi106.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35603.log.gz 2024-04-04T21:51:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:gzip/var/log/ceph/ceph-client.admin.42368.log: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-04T21:51:09.279 INFO:teuthology.orchestra.run.smithi106.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40392.log.gz 2024-04-04T21:51:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35195.log 2024-04-04T21:51:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43810.log 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40816.log.gz 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.35195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39650.log 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40522.log 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39650.log: 90.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41397.log 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42111.log.gz 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.35195.log.gz 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39959.log 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39650.log.gz 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41232.log 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40900.log 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40522.log.gz 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39959.log: /var/log/ceph/ceph-client.admin.41232.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39959.log.gz 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42124.log 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72101.log 2024-04-04T21:51:09.280 INFO:teuthology.orchestra.run.smithi106.stderr: 87.8% 90.8% -- replaced with /var/log/ceph/ceph-client.admin.43810.log.gz 2024-04-04T21:51:09.281 INFO:teuthology.orchestra.run.smithi106.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40900.log.gz/var/log/ceph/ceph-client.admin.42124.log: 2024-04-04T21:51:09.283 INFO:teuthology.orchestra.run.smithi106.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.42124.log.gz 2024-04-04T21:51:09.283 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40306.log 2024-04-04T21:51:09.283 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.72101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40086.log 2024-04-04T21:51:09.283 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40306.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.42368.log.gz 2024-04-04T21:51:09.283 INFO:teuthology.orchestra.run.smithi106.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40306.log.gz 2024-04-04T21:51:09.287 INFO:teuthology.orchestra.run.smithi106.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41232.log.gz 2024-04-04T21:51:09.287 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-04-04T21:51:09.288 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40086.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-04-04T21:51:09.288 INFO:teuthology.orchestra.run.smithi106.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.72101.log.gz 2024-04-04T21:51:09.288 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-04-04T21:51:09.288 INFO:teuthology.orchestra.run.smithi106.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40086.log.gz 2024-04-04T21:51:09.288 INFO:teuthology.orchestra.run.smithi106.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-04-04T21:51:09.288 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-04-04T21:51:09.288 INFO:teuthology.orchestra.run.smithi106.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-04-04T21:51:09.288 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-04-04T21:51:09.288 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/valgrind/osd.0.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-04-04T21:51:09.288 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-04-04T21:51:09.288 INFO:teuthology.orchestra.run.smithi106.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-04-04T21:51:09.288 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41380.log 2024-04-04T21:51:09.288 INFO:teuthology.orchestra.run.smithi106.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-04-04T21:51:09.288 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41244.log 2024-04-04T21:51:09.289 INFO:teuthology.orchestra.run.smithi106.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-04-04T21:51:09.289 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40858.log 2024-04-04T21:51:09.289 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39645.log 2024-04-04T21:51:09.289 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39306.log 2024-04-04T21:51:09.289 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60545.log 2024-04-04T21:51:09.289 INFO:teuthology.orchestra.run.smithi106.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40858.log.gz 2024-04-04T21:51:09.289 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39306.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39645.log.gz 2024-04-04T21:51:09.301 INFO:teuthology.orchestra.run.smithi106.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39306.log.gz 2024-04-04T21:51:09.301 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39870.log 2024-04-04T21:51:09.301 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.60545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39654.log 2024-04-04T21:51:09.301 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40030.log 2024-04-04T21:51:09.302 INFO:teuthology.orchestra.run.smithi106.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39870.log.gz 2024-04-04T21:51:09.302 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39654.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39654.log.gz 2024-04-04T21:51:09.302 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60547.log 2024-04-04T21:51:09.302 INFO:teuthology.orchestra.run.smithi106.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.41380.log.gz 2024-04-04T21:51:09.302 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109115.log 2024-04-04T21:51:09.302 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.60547.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.60545.log.gz 2024-04-04T21:51:09.302 INFO:teuthology.orchestra.run.smithi106.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40030.log.gz 2024-04-04T21:51:09.302 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41853.log 2024-04-04T21:51:09.302 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29317.log 2024-04-04T21:51:09.302 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.109115.log: /var/log/ceph/ceph-client.admin.41853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60548.log 2024-04-04T21:51:09.302 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph.tmp-client.admin.29317.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29317.log.gz 2024-04-04T21:51:09.302 INFO:teuthology.orchestra.run.smithi106.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109115.log.gz 2024-04-04T21:51:09.302 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42187.log 2024-04-04T21:51:09.302 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41853.log.gz 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr: -- /var/log/ceph/ceph-client.admin.39643.log 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.42187.log: /var/log/ceph/ceph-client.admin.60548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53420.log 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39643.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.41244.log.gz 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.60547.log.gz 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39643.log.gz 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42203.log 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.53420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35498.log 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.42203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42292.log 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.35498.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35498.log.gz 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39652.log 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.42292.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.53420.log.gz 90.4% -- replaced with /var/log/ceph/ceph-client.admin.60548.log.gz 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40186.log 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41539.log 2024-04-04T21:51:09.303 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40186.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39652.log.gz 2024-04-04T21:51:09.304 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39472.log 2024-04-04T21:51:09.304 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41539.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40186.log.gz 2024-04-04T21:51:09.304 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108863.log 2024-04-04T21:51:09.304 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39472.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39472.log.gz 2024-04-04T21:51:09.304 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-04T21:51:09.312 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.108863.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108863.log.gz 2024-04-04T21:51:09.312 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42092.log 2024-04-04T21:51:09.312 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph.audit.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.41539.log.gz 2024-04-04T21:51:09.312 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41893.log 2024-04-04T21:51:09.312 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.42092.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.42187.log.gz 2024-04-04T21:51:09.312 INFO:teuthology.orchestra.run.smithi106.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.42092.log.gz 2024-04-04T21:51:09.312 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40618.log 2024-04-04T21:51:09.312 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109325.log 2024-04-04T21:51:09.312 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40618.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40618.log.gz 2024-04-04T21:51:09.312 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108653.log 2024-04-04T21:51:09.312 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.109325.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109325.log.gz 2024-04-04T21:51:09.312 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40404.log 2024-04-04T21:51:09.313 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.108653.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.108653.log.gz 2024-04-04T21:51:09.313 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39119.log 2024-04-04T21:51:09.313 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40404.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40404.log.gz 2024-04-04T21:51:09.317 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-04T21:51:09.317 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35171.log 2024-04-04T21:51:09.317 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-04-04T21:51:09.318 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35171.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-04-04T21:51:09.318 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-04-04T21:51:09.318 INFO:teuthology.orchestra.run.smithi203.stderr: 55.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-04-04T21:51:09.318 INFO:teuthology.orchestra.run.smithi203.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35171.log.gz 2024-04-04T21:51:09.318 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-04-04T21:51:09.318 INFO:teuthology.orchestra.run.smithi203.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-04-04T21:51:09.318 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-04-04T21:51:09.318 INFO:teuthology.orchestra.run.smithi203.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-04-04T21:51:09.318 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-04T21:51:09.318 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/valgrind/mon.b.log: /var/log/ceph/valgrind/osd.4.log: 69.5% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-04-04T21:51:09.318 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35051.log 2024-04-04T21:51:09.318 INFO:teuthology.orchestra.run.smithi203.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-04-04T21:51:09.318 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35091.log 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.4.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35051.log.gz 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35091.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.5.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35091.log.gz 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35215.log 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34971.log 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35215.log: /var/log/ceph/ceph-mgr.x.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35215.log.gz 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr: 92.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35011.log 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34971.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35011.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34971.log.gz 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35131.log 2024-04-04T21:51:09.319 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35011.log.gz 2024-04-04T21:51:09.325 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37602.log 2024-04-04T21:51:09.325 INFO:teuthology.orchestra.run.smithi106.stderr: 94.4% -- replaced with /var/log/ceph/ceph-client.admin.42292.log.gz 2024-04-04T21:51:09.325 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39119.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39119.log.gz 2024-04-04T21:51:09.325 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41739.log 2024-04-04T21:51:09.325 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.37602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38868.log 2024-04-04T21:51:09.325 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41739.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37602.log.gz 2024-04-04T21:51:09.326 INFO:teuthology.orchestra.run.smithi106.stderr: 95.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-04T21:51:09.326 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72103.log 2024-04-04T21:51:09.326 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.38868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41703.log 2024-04-04T21:51:09.326 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.72103.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38868.log.gz 2024-04-04T21:51:09.326 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41631.log 2024-04-04T21:51:09.326 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41703.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41739.log.gz 2024-04-04T21:51:09.326 INFO:teuthology.orchestra.run.smithi106.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.72103.log.gz 2024-04-04T21:51:09.326 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42263.log 2024-04-04T21:51:09.326 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40642.log 2024-04-04T21:51:09.328 INFO:teuthology.orchestra.run.smithi203.stderr: 88.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-04T21:51:09.328 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35262.log 2024-04-04T21:51:09.328 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35131.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-04T21:51:09.328 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35262.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35131.log.gz 2024-04-04T21:51:09.328 INFO:teuthology.orchestra.run.smithi203.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35262.log.gz 2024-04-04T21:51:09.329 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41631.log: /var/log/ceph/ceph-client.admin.42263.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.42203.log.gz 2024-04-04T21:51:09.331 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39250.log 2024-04-04T21:51:09.332 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40642.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40642.log.gz 2024-04-04T21:51:09.334 INFO:teuthology.orchestra.run.smithi106.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41631.log.gz 2024-04-04T21:51:09.336 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108737.log 2024-04-04T21:51:09.336 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39648.log 2024-04-04T21:51:09.336 INFO:teuthology.orchestra.run.smithi106.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39250.log.gz 2024-04-04T21:51:09.336 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.108737.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108737.log.gz 2024-04-04T21:51:09.338 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60549.log 2024-04-04T21:51:09.338 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43470.log 2024-04-04T21:51:09.338 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.60549.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39648.log.gz 2024-04-04T21:51:09.338 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41915.log 2024-04-04T21:51:09.340 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.b.log: 95.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-04T21:51:09.340 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43470.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.60549.log.gz 2024-04-04T21:51:09.343 INFO:teuthology.orchestra.run.smithi106.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.43470.log.gz 2024-04-04T21:51:09.344 INFO:teuthology.orchestra.run.smithi106.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.admin.41703.log.gz 2024-04-04T21:51:09.344 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108779.log 2024-04-04T21:51:09.344 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36584.log 2024-04-04T21:51:09.344 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.108779.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-04T21:51:09.344 INFO:teuthology.orchestra.run.smithi106.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41915.log.gz 2024-04-04T21:51:09.344 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.36584.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108779.log.gz 2024-04-04T21:51:09.345 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41261.log 2024-04-04T21:51:09.345 INFO:teuthology.orchestra.run.smithi106.stderr: 88.7%/var/log/ceph/ceph-osd.1.log: -- replaced with /var/log/ceph/ceph-client.admin.36584.log.gz 2024-04-04T21:51:09.345 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39894.log 2024-04-04T21:51:09.345 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109031.log 2024-04-04T21:51:09.346 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39894.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39894.log.gz 2024-04-04T21:51:09.346 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41184.log 2024-04-04T21:51:09.347 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.109031.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.41261.log.gz 2024-04-04T21:51:09.347 INFO:teuthology.orchestra.run.smithi106.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109031.log.gz 2024-04-04T21:51:09.347 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108695.log 2024-04-04T21:51:09.347 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40562.log 2024-04-04T21:51:09.348 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41184.log: /var/log/ceph/ceph-client.admin.108695.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108695.log.gz 2024-04-04T21:51:09.352 INFO:teuthology.orchestra.run.smithi106.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.admin.41351.log.gz 2024-04-04T21:51:09.358 INFO:teuthology.orchestra.run.smithi106.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.42263.log.gz 2024-04-04T21:51:09.361 INFO:teuthology.orchestra.run.smithi106.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.41893.log.gz 2024-04-04T21:51:09.361 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39651.log 2024-04-04T21:51:09.361 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41271.log 2024-04-04T21:51:09.365 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39388.log 2024-04-04T21:51:09.365 INFO:teuthology.orchestra.run.smithi106.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40562.log.gz 2024-04-04T21:51:09.365 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40622.log 2024-04-04T21:51:09.365 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41271.log: /var/log/ceph/ceph-client.admin.39388.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39651.log.gz 2024-04-04T21:51:09.365 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39430.log 2024-04-04T21:51:09.365 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40763.log 2024-04-04T21:51:09.365 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39430.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39388.log.gz 2024-04-04T21:51:09.365 INFO:teuthology.orchestra.run.smithi106.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40622.log.gz 2024-04-04T21:51:09.365 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41693.log 2024-04-04T21:51:09.365 INFO:teuthology.orchestra.run.smithi106.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39430.log.gz 2024-04-04T21:51:09.365 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108905.log 2024-04-04T21:51:09.366 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41827.log 2024-04-04T21:51:09.366 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.108905.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40763.log.gz 2024-04-04T21:51:09.366 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108527.log 2024-04-04T21:51:09.366 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41827.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.108905.log.gz 2024-04-04T21:51:09.366 INFO:teuthology.orchestra.run.smithi106.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.41827.log.gz 2024-04-04T21:51:09.368 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42097.log 2024-04-04T21:51:09.368 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.108527.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-04T21:51:09.368 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.42097.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108527.log.gz 2024-04-04T21:51:09.370 INFO:teuthology.orchestra.run.smithi106.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.41693.log.gz 2024-04-04T21:51:09.374 INFO:teuthology.orchestra.run.smithi106.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.42097.log.gz 2024-04-04T21:51:09.374 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39909.log 2024-04-04T21:51:09.375 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42195.log 2024-04-04T21:51:09.375 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43811.log 2024-04-04T21:51:09.376 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.42195.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39909.log.gz 2024-04-04T21:51:09.388 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53421.log 2024-04-04T21:51:09.391 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43811.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.43811.log.gz 2024-04-04T21:51:09.392 INFO:teuthology.orchestra.run.smithi106.stderr: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.42195.log.gz 2024-04-04T21:51:09.396 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35734.log 2024-04-04T21:51:09.396 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.53421.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.41184.log.gz 2024-04-04T21:51:09.396 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41604.log 2024-04-04T21:51:09.396 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40286.log 2024-04-04T21:51:09.396 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.35734.log: /var/log/ceph/ceph-client.admin.41604.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35734.log.gz 2024-04-04T21:51:09.396 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41940.log 2024-04-04T21:51:09.397 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40286.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.53421.log.gz 2024-04-04T21:51:09.397 INFO:teuthology.orchestra.run.smithi106.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40286.log.gz 2024-04-04T21:51:09.397 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39874.log 2024-04-04T21:51:09.397 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42393.log 2024-04-04T21:51:09.399 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41940.log: /var/log/ceph/ceph-client.admin.39874.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39874.log.gz 2024-04-04T21:51:09.399 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40206.log 2024-04-04T21:51:09.400 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.42393.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.41940.log.gz 2024-04-04T21:51:09.405 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39939.log 2024-04-04T21:51:09.405 INFO:teuthology.orchestra.run.smithi106.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.42393.log.gz 2024-04-04T21:51:09.405 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40206.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.41604.log.gz 2024-04-04T21:51:09.405 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108989.log 2024-04-04T21:51:09.406 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109199.log 2024-04-04T21:51:09.406 INFO:teuthology.orchestra.run.smithi106.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40206.log.gz 2024-04-04T21:51:09.406 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.108989.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39939.log.gz 2024-04-04T21:51:09.406 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42226.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.108989.log.gz 2024-04-04T21:51:09.406 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:51:09.406 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.109199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53419.log 2024-04-04T21:51:09.406 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.42226.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109199.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-04T21:51:09.406 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:51:09.406 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60546.log 2024-04-04T21:51:09.406 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.53419.log: /var/log/ceph/ceph-mgr.y.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42226.log.gz 2024-04-04T21:51:09.406 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108821.log 2024-04-04T21:51:09.408 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.60546.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.53419.log.gz 2024-04-04T21:51:09.408 INFO:teuthology.orchestra.run.smithi106.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.60546.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41785.log 2024-04-04T21:51:09.408 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:51:09.408 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43471.log 2024-04-04T21:51:09.408 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.108821.log: /var/log/ceph/ceph-client.admin.41785.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108821.log.gz 2024-04-04T21:51:09.409 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40285.log 2024-04-04T21:51:09.410 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43471.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.43471.log.gz 2024-04-04T21:51:09.410 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41865.log 2024-04-04T21:51:09.411 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40285.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40285.log.gz 2024-04-04T21:51:09.411 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108569.log 2024-04-04T21:51:09.417 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41865.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41785.log.gz 2024-04-04T21:51:09.421 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43538.log 2024-04-04T21:51:09.421 INFO:teuthology.orchestra.run.smithi106.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.41865.log.gz/var/log/ceph/ceph-client.admin.108569.log: 2024-04-04T21:51:09.421 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41404.log 2024-04-04T21:51:09.421 INFO:teuthology.orchestra.run.smithi106.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108569.log.gz 2024-04-04T21:51:09.421 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42114.log 2024-04-04T21:51:09.421 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41404.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43538.log.gz 2024-04-04T21:51:09.421 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109283.log 2024-04-04T21:51:09.422 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.42114.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42114.log.gz 2024-04-04T21:51:09.422 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39346.log 2024-04-04T21:51:09.423 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.109283.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109283.log.gz 2024-04-04T21:51:09.423 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108611.log 2024-04-04T21:51:09.424 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39346.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.41271.log.gz 2024-04-04T21:51:09.424 INFO:teuthology.orchestra.run.smithi106.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39346.log.gz 2024-04-04T21:51:09.426 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41941.log 2024-04-04T21:51:09.426 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5/var/log/ceph/ceph-client.admin.108611.log: --verbose -- /var/log/ceph/ceph-client.admin.40457.log 2024-04-04T21:51:09.426 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41941.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108611.log.gz 2024-04-04T21:51:09.426 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42400.log 2024-04-04T21:51:09.427 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40457.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40457.log.gz 2024-04-04T21:51:09.428 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-04T21:51:09.428 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.42400.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.41941.log.gz 2024-04-04T21:51:09.429 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40723.log 2024-04-04T21:51:09.442 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109368.log 2024-04-04T21:51:09.443 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.40723.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40723.log.gz 2024-04-04T21:51:09.449 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41566.log 2024-04-04T21:51:09.462 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.109368.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109241.log 2024-04-04T21:51:09.462 INFO:teuthology.orchestra.run.smithi106.stderr: -- replaced with /var/log/ceph/ceph-client.admin.109368.log.gz 2024-04-04T21:51:09.469 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41566.log: 91.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41594.log 2024-04-04T21:51:09.477 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.109241.log: 87.9% 92.2% -- replaced with /var/log/ceph/ceph-client.admin.42400.log.gz 2024-04-04T21:51:09.477 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42310.log 2024-04-04T21:51:09.477 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41594.log: -- replaced with /var/log/ceph/ceph-client.admin.109241.log.gz 2024-04-04T21:51:09.478 INFO:teuthology.orchestra.run.smithi106.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41566.log.gz 2024-04-04T21:51:09.481 INFO:teuthology.orchestra.run.smithi106.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.41594.log.gz 2024-04-04T21:51:09.481 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41516.log 2024-04-04T21:51:09.486 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.42310.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.42310.log.gz 2024-04-04T21:51:09.486 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-04T21:51:09.494 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41516.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.41516.log.gz 2024-04-04T21:51:09.494 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41305.log 2024-04-04T21:51:09.507 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-04T21:51:09.516 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41305.log: 91.6% 93.5% -- replaced with /var/log/ceph/ceph-client.admin.41305.log.gz 2024-04-04T21:51:09.516 INFO:teuthology.orchestra.run.smithi106.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-04-04T21:51:09.516 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43812.log 2024-04-04T21:51:09.534 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41203.log 2024-04-04T21:51:09.538 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43812.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.43812.log.gz 2024-04-04T21:51:09.551 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39209.log 2024-04-04T21:51:09.569 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.41203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109450.log 2024-04-04T21:51:09.569 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39209.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39209.log.gz 2024-04-04T21:51:09.575 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43537.log 2024-04-04T21:51:09.582 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.109450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43472.log 2024-04-04T21:51:09.582 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43537.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109450.log.gz 2024-04-04T21:51:09.585 INFO:teuthology.orchestra.run.smithi106.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.43537.log.gz 2024-04-04T21:51:09.589 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109073.log 2024-04-04T21:51:09.598 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43472.log: 91.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39647.log 2024-04-04T21:51:09.605 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.109073.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43472.log.gz 2024-04-04T21:51:09.607 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72102.log 2024-04-04T21:51:09.609 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.39647.log: -- replaced with /var/log/ceph/ceph-client.admin.109073.log.gz 2024-04-04T21:51:09.614 INFO:teuthology.orchestra.run.smithi106.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39647.log.gz 2024-04-04T21:51:09.626 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42125.log 2024-04-04T21:51:09.629 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.72102.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.72102.log.gz 2024-04-04T21:51:09.642 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42365.log 2024-04-04T21:51:09.643 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.42125.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42125.log.gz 2024-04-04T21:51:09.658 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109410.log 2024-04-04T21:51:09.674 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.42365.log: 90.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109157.log 2024-04-04T21:51:09.675 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.109410.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109410.log.gz 2024-04-04T21:51:09.676 INFO:teuthology.orchestra.run.smithi106.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.41203.log.gz 2024-04-04T21:51:09.684 INFO:teuthology.orchestra.run.smithi106.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42365.log.gz 2024-04-04T21:51:09.689 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-04T21:51:09.690 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.109157.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.109157.log.gz 2024-04-04T21:51:09.949 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-mon.c.log: 93.1% -- replaced with /var/log/ceph/ceph-client.admin.41404.log.gz 2024-04-04T21:51:10.074 INFO:teuthology.orchestra.run.smithi106.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-04T21:51:10.665 INFO:teuthology.orchestra.run.smithi106.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41397.log.gz 2024-04-04T21:51:10.822 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-04-04T21:51:11.289 INFO:teuthology.orchestra.run.smithi106.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-04T21:51:11.680 INFO:teuthology.orchestra.run.smithi203.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-04T21:51:13.860 INFO:teuthology.orchestra.run.smithi106.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-04T21:51:15.014 INFO:teuthology.orchestra.run.smithi203.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-04T21:51:15.573 INFO:teuthology.orchestra.run.smithi106.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-04T21:51:15.823 INFO:teuthology.orchestra.run.smithi106.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-04T21:51:15.857 INFO:teuthology.orchestra.run.smithi203.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-04T21:51:16.182 INFO:teuthology.orchestra.run.smithi106.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-04T21:51:16.582 INFO:teuthology.orchestra.run.smithi106.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-04T21:51:16.584 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-04T21:51:16.584 INFO:teuthology.orchestra.run.smithi106.stderr:real 0m7.325s 2024-04-04T21:51:16.584 INFO:teuthology.orchestra.run.smithi106.stderr:user 0m34.479s 2024-04-04T21:51:16.584 INFO:teuthology.orchestra.run.smithi106.stderr:sys 0m1.224s 2024-04-04T21:51:17.275 INFO:teuthology.orchestra.run.smithi203.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-04T21:51:17.430 INFO:teuthology.orchestra.run.smithi203.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-04T21:51:17.432 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-04T21:51:17.432 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m8.143s 2024-04-04T21:51:17.432 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m29.871s 2024-04-04T21:51:17.432 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m0.880s 2024-04-04T21:51:17.432 INFO:tasks.ceph:Archiving logs... 2024-04-04T21:51:17.433 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121/remote/smithi106/log 2024-04-04T21:51:17.434 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-04T21:51:22.236 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121/remote/smithi203/log 2024-04-04T21:51:22.237 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-04T21:51:26.147 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-04-04T21:51:26.147 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-04T21:51:26.156 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_66deef1598934c3b1ef2f044e380a7d00244cead/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-04-04T21:51:26.157 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-04T21:51:26.157 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-04T21:51:26.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-04T21:51:26.231 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-04T21:51:26.240 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-04T21:51:26.240 DEBUG:teuthology.orchestra.run.smithi106:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-04T21:51:26.243 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-04T21:51:26.257 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-04-04T21:51:26.257 INFO:teuthology.orchestra.run.smithi106.stderr:bash: line 1: ntpq: command not found 2024-04-04T21:51:26.261 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-04T21:51:26.261 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-04-04T21:51:26.262 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 -774us[ -787us] +/- 83ms 2024-04-04T21:51:26.262 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 +91us[ +77us] +/- 60ms 2024-04-04T21:51:26.262 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +681us[ +667us] +/- 88ms 2024-04-04T21:51:26.262 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-04T21:51:26.263 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-04T21:51:26.263 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-04-04T21:51:26.263 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 21 -756us[ -752us] +/- 83ms 2024-04-04T21:51:26.263 INFO:teuthology.orchestra.run.smithi106.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 +89us[ +93us] +/- 60ms 2024-04-04T21:51:26.263 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +676us[ +680us] +/- 88ms 2024-04-04T21:51:26.263 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-04T21:51:26.263 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-04T21:51:26.273 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-04T21:51:26.273 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-04T21:51:26.282 DEBUG:teuthology.orchestra.run.smithi106:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-04T21:51:26.374 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-04T21:51:26.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-04T21:51:26.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-04T21:51:26.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-04T21:51:26.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-04T21:51:26.500 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-04T21:51:26.510 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-04T21:51:26.519 INFO:teuthology.task.internal:Duration was 3966.716377 seconds 2024-04-04T21:51:26.519 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-04T21:51:26.529 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-04T21:51:26.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-04T21:51:26.533 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-04T21:51:26.575 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-04T21:51:26.584 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-04T21:51:26.792 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-04T21:51:26.792 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi106.front.sepia.ceph.com 2024-04-04T21:51:26.792 DEBUG:teuthology.orchestra.run.smithi106:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-04T21:51:26.811 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-04-04T21:51:26.811 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-04T21:51:26.842 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-04T21:51:26.842 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-04T21:51:26.853 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-04T21:51:26.910 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-04T21:51:26.910 DEBUG:teuthology.orchestra.run.smithi106:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-04T21:51:26.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-04T21:51:27.054 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-04T21:51:27.065 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-04T21:51:27.065 DEBUG:teuthology.orchestra.run.smithi106:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-04T21:51:27.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-04T21:51:27.120 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-04T21:51:27.130 DEBUG:teuthology.orchestra.run.smithi106:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-04T21:51:27.135 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-04T21:51:27.158 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = core 2024-04-04T21:51:27.185 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-04-04T21:51:27.211 DEBUG:teuthology.orchestra.run.smithi106:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-04T21:51:27.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-04T21:51:27.229 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-04T21:51:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-04T21:51:27.265 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-04T21:51:27.275 INFO:teuthology.task.internal:Transferring archived files... 2024-04-04T21:51:27.277 DEBUG:teuthology.misc:Transferring archived files from smithi106:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121/remote/smithi106 2024-04-04T21:51:27.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-04T21:51:27.334 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121/remote/smithi203 2024-04-04T21:51:27.334 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-04T21:51:27.369 INFO:teuthology.task.internal:Removing archive directory... 2024-04-04T21:51:27.369 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-04T21:51:27.372 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-04T21:51:27.418 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-04T21:51:27.431 INFO:teuthology.task.internal:Not uploading archives. 2024-04-04T21:51:27.431 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-04T21:51:27.441 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-04T21:51:27.441 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-04T21:51:27.444 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-04T21:51:27.455 INFO:teuthology.orchestra.run.smithi106.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 4 21:51 /home/ubuntu/cephtest 2024-04-04T21:51:27.471 INFO:teuthology.orchestra.run.smithi203.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 4 21:51 /home/ubuntu/cephtest 2024-04-04T21:51:27.472 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-04T21:51:27.483 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-04T21:51:27.493 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-04-04T21:51:27.493 INFO:teuthology.nuke:Checking targets against current locks 2024-04-04T21:51:27.520 DEBUG:teuthology.nuke:shortname: smithi106 2024-04-04T21:51:27.521 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-04T21:51:27.545 DEBUG:teuthology.nuke:shortname: smithi203 2024-04-04T21:51:27.546 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-04T21:51:27.550 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-04 20:36:01.496276', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO5pKSz3FNac+x2k6GP1zszcrUyY03hsljHicGucMoj4mXJbN0r0LR7iGE+oQekIZlxoMd2ak7lpFdpiLOTaUQQ='} 2024-04-04T21:51:27.570 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-04 20:36:01.495100', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBqwBEST0LdPtiFBL4rbRllWhJoAZHrBC6yTmDPvJRG/iuIvA4abrIenfHOl7SxciGjcSoBIQe6fuYuK6YSMIqA='} 2024-04-04T21:51:27.599 INFO:teuthology.orchestra.console.smithi106:Power off 2024-04-04T21:51:27.599 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-04T21:51:27.623 DEBUG:teuthology.orchestra.console.smithi106:power off output: Chassis Power Control: Down/Off 2024-04-04T21:51:27.623 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-04T21:51:27.637 INFO:teuthology.orchestra.console.smithi203:Power off 2024-04-04T21:51:27.637 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-04T21:51:27.646 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-04-04T21:51:27.660 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-04-04T21:51:27.660 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-04T21:51:27.684 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-04-04T21:51:31.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-04-04T21:51:31.648 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-04T21:51:31.685 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-04T21:51:31.758 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-04-04T21:51:31.795 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-04-04T21:51:35.762 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-04T21:51:35.795 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-04T21:51:35.872 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is off 2024-04-04T21:51:35.905 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-04-04T21:51:35.974 INFO:teuthology.orchestra.console.smithi106:Power off completed 2024-04-04T21:51:36.006 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-04-04T21:51:36.111 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} duration: 3966.716376543045 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-04-04T21:51:36.111 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-04T21:51:36.160 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~5s 2024-04-04T21:51:36.193 INFO:teuthology.run:FAIL